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U.S. Northern Command 
(NORTHCOM) exercises to test 
preparedness to perform its 
homeland defense and civil support 
missions. GAO was asked to assess 
the extent to which NORTHCOM is 
(1) consistent with Department of 
Defense (DOD) training and 
exercise requirements, (2) 
involving interagency partners and 
states in its exercises, (3) using 
lessons learned and corrective 
actions to improve preparedness, 
and (4) integrating its exercises 
with the National Exercise 
Program (NEP). To do this, GAO 
reviewed NORTHCOM and NEP 
guidance and postexercise 
documentation, assessed 
NORTHCOM compliance, and 
compared DOD and NEP exercise 
requirements. 
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GAO is making recommendations 
to DOD to direct NORTHCOM to 
consistently involve the states in 
planning, executing, and assessing 
exercises and improve oversight of 
corrective actions. GAO is also 
recommending that DOD define 
when NORTHCOM should use NEP 
planning and documentation 
requirements. DOD agreed or 
partially agreed with the 
recommendations and cited 
ongoing and future efforts to 
satisfy the recommendations' 
intent. DOD did not fully address a 
recommendation on training to 
NORTHCOM staff on specific state 
emergency management structures. 
GAO believes such training would 
benefit NORTHCOM personnel in 
advance of a crisis and for exercise 
planning. 
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What GAO Found 

NORTHCOM's exercise program is generally consistent with the requirements 
of DOD's Joint Training System, but its exercise reporting is inconsistent. 
Since the command was established in 2002, NORTHCOM has conducted 13 
large-scale exercises and generally completed exercise summary reports 
within the required time frame. However, those reports did not consistently 
include certain information, such as areas needing improvement, because 
NORTHCOM lacks guidance that specifies exercise reports' content and 
format, potentially impacting its ability to meet internal standards for planning 
and execution of joint exercises, and to compare and share exercise results 
over time with interagency partners and states. 

Nineteen federal agencies and organizations and 17 states and the District of 
Columbia have participated in one or more of the seven large-scale exercises 
that NORTHCOM has conducted since September 2005. However, 
NORTHCOM faces challenges in involving states in the planning, conduct, and 
assessment of its exercises, such as adapting its exercise system and practices 
to involve other federal, state, local, and tribal agencies that do not have the 
same practices or level of planning resources. Inconsistencies with how 
NORTHCOM involves states in exercises are occurring in part because 
NORTHCOM officials lack experience dealing with states and do not have a 
consistent process for including states in exercises. Without such a process, 
NORTHCOM increases the risk that its exercises will not provide benefits for 
all participants, impact the seamless exercise of all levels of government, and 
potentially affect NORTHCOM's ability to provide civil support capabilities. 

NORTHCOM has a systematic lessons learned and corrective action program 
to improve preparedness, but gaps remain with collecting and sharing lessons 
with agency and state partners and managing corrective actions. Access to the 
system NORTHCOM uses for managing exercise observations is limited for 
non-DOD participants, and DOD believes that the Department of Homeland 
Security's system is not adequately protected from unauthorized users. 
NORTHCOM's mitigation steps have not resolved the issues. In addition, 
about 20 percent of the corrective actions tracked by NORTHCOM were being 
closed prematurely due to gaps in oversight. Closing issues prematurely 
increases the risk that issues will reoccur and limits the knowledge gained and 
value of the exercise. 

NORTHCOM has taken steps to integrate its exercises with the NEP, but 
guidance is not consistently applied. NORTHCOM has participated in several 
NEP exercises and is leading its first major NEP exercise in the fall of 2009. 
However, NORTHCOM has used DOD's Joint Training System planning and 
documentation requirements rather than DHS's requirements, because NEP 
guidance is not clear on what exercise planning standard should be used and 
DOD guidance does not address the issue. The states we visited use NEP 
guidance. Differences between NEP and DOD guidance could affect the ability 
of all participants to develop effective working relationships. 
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September 9, 2009 
Congressional Requesters 

The U.S. homeland continues to face complex and dynamic threats and 
vulnerabilities from terrorism and other catastrophic incidents, including 
natural disasters, industrial accidents, infrastructure failures, and 
infectious diseases. 1 These incidents can produce devastating 
consequences as observed during the September 11, 2001, terrorist attacks 
and in the aftermath of Hurricane Katrina, which required an effective and 
coordinated national effort, with shared goals and responsibilities for 
protecting and defending the homeland. The complexity of national-level 
coordination makes preparing for an incident all the more crucial. 2 
According the Quadrennial Defense Review, unified interagency efforts 
among the Department of Defense (DOD), Department of Homeland 
Security (DHS), and other federal, state, and local agencies are required to 
address threats to the homeland. 3 Exercises play an instrumental role in 
preparing these agencies to respond to an incident by providing 
opportunities to test plans, improve proficiency, assess capabilities and 
readiness, and clarify roles and responsibilities. 4 Short of performance in 
actual operations, exercises provide the best means to assess the 
effectiveness of organizations in achieving mission preparedness. 
Exercises also provide an ideal opportunity to enhance preparedness by 
collecting, developing, implementing, and disseminating lessons learned 
and verifying corrective action taken to resolve previously identified 
issues. 5 



homeland Security Council, National Strategy for Homeland Security (Washington, D.C: 
October 2007), 1, 9, 13. 

2 GAO, National Preparedness: FEMA Has Made Progress, but Needs to Complete and 
Integrate Planning, Exercise, and Assessment Efforts, GAO-09-369 (Washington, D.C: 
Apr. 30, 2009). 

3 DOD, Quadrennial Defense Review Report (Washington, D.C: Feb. 6, 2006), 87. 

4 An exercise is a training event where scenarios are enacted to assess training proficiency 
and identify lessons learned to improve preparedness. Exercises range from tabletop 
computer simulations to the full-scale deployment of personnel and equipment. 

°U.S. Northern Command, Commander's Training Guidance for FY09- 10 (Colorado 
Springs, Colo.: Apr. 8, 2008), A-20. 
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Following the September 11 terrorist attacks, DOD realized the need for a 
more integrated civilian and military response capability for any future 
attack on the homeland. In response, DOD established the U.S. Northern 
Command (NORTHCOM) to provide and manage homeland defense and 
civil support. These two activities are among DOD's contribution to 
homeland security. DHS is the lead federal agency for homeland security, 
which is a national effort to prevent terrorist attacks within the United 
States, reduce America's vulnerability to terrorism, and minimize the 
damage and recover from attacks that do occur. 6 DOD is the lead federal 
agency for homeland defense — defined as the protection of U.S. 
sovereignty, territory, domestic population, and critical defense 
infrastructure against external threats and aggression against the United 
States. 7 Civil support is defense support of civil authorities — such as DHS 
or another federal agency or state — for domestic emergencies and for 
designated law enforcement and other activities. 8 NORTHCOM differs 
from other combatant commands in that, in addition to Canada and 
Mexico, its area of responsibility includes the 48 contiguous states, Alaska, 
the District of Columbia, Puerto Rico, and the U.S. Virgin Islands. 

Numerous local, state, tribal, and federal agencies and organizations — 
including DOD, DHS, and the Federal Emergency Management Agency 
(FEMA) — have jurisdiction over or can coordinate resources within the 
homeland and, therefore, may be involved in the response to an incident. 
According to the National Defense Strategy, a whole-of-government 
approach is only possible when every government department and agency 
understands the core competencies, roles, missions, and capabilities of its 
partners and works together to achieve common goals. 9 Ensuring an 
effective response to an incident will require that federal departments and 
agencies, states, and local governments conduct integrated disaster 
response planning and test these plans by exercising together. 10 To 
achieve its goal of being ready to execute joint operations and ensure a 
seamless operating environment, NORTHCOM's training goals include 
maximizing exercise participation with federal, state, and local agencies 



6 DOD, Strategy for Homeland Defense and Civil Support, 5. 

7 DOD, Strategy for Homeland Defense and Civil Support, 5. 

8 DOD, Strategy for Homeland Defense and Civil Support, 5-6. 

9 DOD, National Defense Strategy (Washington, D.C.: June 2008), 17-18. 

10 Defense Science Board, Unconventional Operational Concepts and the Homeland 
(Washington, D.C.: March 2009), 37, 48, 50. 
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and National Guard units. 11 NORTHCOM uses long-established DOD 
practices for training and exercises to test and enhance the preparedness 
of its personnel and forces to perform its homeland defense and civil 
support missions. As a result, DOD has more experience than other federal 
agencies in terms of the scope and range of its exercises and the 
robustness of the process for conducting them. DOD's goal is to improve 
the homeland defense and consequence management capabilities of its 
interagency partners by leveraging this comparative advantage through 
shared training and exercises. 12 

The National Exercise Program (NEP) was established in April 2007 under 
the leadership of the Secretary of Homeland Security to prioritize and 
coordinate the various federal, regional, and state exercise activities and 
serves as the principal mechanism for examining the preparation and 
efficiency of the federal government to respond to an incident. 13 The NEP 
includes a series of national exercises projected on a 5-year exercise 
schedule and has established a four-tier system to determine the relative 
priority of interagency participation in each exercise. DOD and other 
federal agencies, states, and local organizations participate in these 
national-level exercises. FEMA administers the NEP and has developed 
Homeland Security Exercise and Evaluation Program guidance to provide 
standardized policy, methodology, and terminology for exercise design, 
development, conduct, evaluation, and improvement planning. 

In our previous work following Hurricanes Katrina and Rita, we identified 
a need to improve, among other things, the nation's disaster response 
capabilities. Overall, capabilities are built upon the appropriate 
combination of people, skills, processes, and assets. Ensuring that needed 
capabilities are available requires effective planning and coordination in 
conjunction with training and exercises in which the capabilities are 
realistically tested and problems identified and subsequently addressed in 
partnership with other federal, state, and local stakeholders. 14 Specific to 



U.S. Northern Command, Commander's Training Guidance for FY09-10, A-4. 
12 DOD, Quadrennial Defense Review Report, 26-27. 

13 DHS, National Exercise Program Implementation Plan (Washington D.C.: Apr. 11, 2007). 
DHS issued a revised Implementation Plan in June 2008. 

14 GAO, Homeland Security: Observations on DHS and FEMA Efforts to Prepare for and 
Respond to Major and Catastrophic Disasters and Address Related Recommendations 
and Legislation, GAO-07-835T (Washington, D.C.: May 15, 2007). 
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DOD, we reported that prior to Hurricane Katrina, disaster plans and 
training exercises involving DOD were insufficient, and did not 
incorporate lessons learned from past catastrophes to fully delineate the 
military capabilities that could be needed to respond to a catastrophic 
natural disaster. Moreover, disaster plans had not been tested and refined 
with a robust exercise program. As a result of the inadequate plans — and 
the lack of realistic exercises to test those plans — a lack of understanding 
existed within DOD and among federal, state, and local responders as to 
the types of assistance and capabilities that DOD might provide, the timing 
of this assistance, and the respective contributions of the active-duty and 
National Guard components. We recommended that DOD establish 
milestones and expedite the development of detailed plans and exercises 
to fully account for the unique capabilities and support that the military is 
likely to provide to civil authorities in response to an incident. 15 In 
addition, we recently found that although FEMA has developed plans, 
guidance, and systems to design exercises, gather lessons learned, and 
track corrective actions to implement the NEP, it faces challenges in 
ensuring that the exercises are carried out in accordance with Homeland 
Security Exercise and Evaluation Program guidance. We recommended, 
among other things, that FEMA establish a program management plan and 
better ensure that exercises follow Homeland Security Exercise 
Evaluation Program guidance, for example, by revising its grant 
monitoring guidance to include a checklist of specific requirements for 
states that receive grant funds. 16 

In order to better understand NORTHCOM's efforts to effectively prepare 
for its homeland defense and civil support missions, we were asked to 
review NORTHCOM's exercise program. This review is the final part of a 
broader congressional request to review NORTHCOM's efforts to plan and 
coordinate its homeland defense and civil support missions. We have 
issued two previous reports in support of that request. 17 For this report, 
our objectives were to determine the extent to which (1) NORTHCOM's 



15 GAO, Hurricane Katrina: Better Plans and Exercises Needed to Guide the Military's 
Response to Catastrophic Natural Disaster's, GAO-06-643 (Washington, D.C.: Mayl6, 2006). 

16 GAO-09-369. 

17 GAO, Homeland Defense: U.S. Northern Command Has Made Progress but Needs to 
Address Force Allocation, Readiness, Tracking Gaps, and Other' Issues, GAO-08-251 
(Washington, D.C.: Apr. 16, 2008); and Homeland Defense: Steps Have Been Taken to 
Improve U.S. Northern Command's Coordination with the States and the National 
Guard Bureau, but Gaps Remain, GAO-08-252 (Washington, D.C.: Apr. 18, 2008). 
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exercise program is consistent with DOD training and exercise 
requirements; (2) NORTHCOM involves relevant interagency partners and 
states in planning, conducting, and assessing exercises; (3) NORTHCOM is 
using lessons learned and corrective actions during exercises to improve 
mission preparedness; and (4) NORTHCOM is integrating its exercises 
with the NEP. 

In conducting this review, we generally focused our scope on 
NORTHCOM's large-scale exercises 18 conducted since Hurricane Katrina 
made landfall in August 2005. To determine the extent to which 
NORTHCOM's exercise program is consistent with DOD exercise 
requirements and includes relevant exercise partners, we reviewed 
exercise documentation for all 13 large-scale exercises the command 
performed since it was established in 2002; evaluated NORTHCOM's 
compliance with requirements established in DOD guidance; and assessed 
the level of interagency and state participation in NORTHCOM's large- 
scale exercises. We also interviewed Office of the Secretary of Defense, 
Joint Staff, NORTHCOM, FEMA, FEMA regional offices, and state officials 
with knowledge of and experience with NORTHCOM's exercise program. 
We selected a nongeneralizable sampling of states based on the extent to 
which they have participated in major NORTHCOM exercises since 
Hurricane Katrina and the varying scenarios of the exercises. The states 
we selected played a major role in NORTHCOM exercises by having a 
portion of the exercise conducted in their state and having various state 
agencies and officials participate. States we selected include Arizona, 
California, Michigan, Oregon, Rhode Island, and Washington. We also met 
with Nevada officials who participated in a NORTHCOM exercise prior to 
Hurricane Katrina — Determined Promise 03 — to provide context to the 
extent that changes may have been made to NORTHCOM's exercise 
program and help develop our state selection methodology. To determine 
the extent to which NORTHCOM is using lessons learned during exercises 
to improve mission preparedness, we reviewed DOD, NORTHCOM, and 
DHS's NEP guidance for lessons learned. Based on this guidance, we 
assessed the management of all unclassified exercise observations and 
issues identified from its last 6 large-scale exercises by determining each 
record's status (open or closed), type (issue or lesson learned), and 
disposition after NORTHCOM staff have acted on these records. Finally, to 



18 Large-scale exercises train and evaluate actual forces deployed in a field environment 
under simulated scenarios. NORTHCOM generally conducts two large-scale exercises 
annually. 
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determine the extent to which NORTHCOM is integrating its exercises 
with the NEP we reviewed DOD and DHS guidance regarding the NEP and 
how DOD and its subordinate commands are required to or should 
participate. We conducted our review between June 2008 and September 
2009 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit to 
obtain sufficient, appropriate evidence to provide a reasonable basis for 
our findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our findings 
and conclusions based on our audit objectives. 



RGSllltS in Brief NORTHCOM's exercise program is generally consistent with the overall 

~ " . policies and practices of DOD's Joint Training System, but the way it 

reports on exercises is inconsistent. DOD and NORTHCOM guidance 
require that the command establish an exercise program consistent with 
the Joint Training System and complete postexercise documentation 
within an established time frame. 19 NORTHCOM has developed a 
comprehensive exercise program consistent with DOD's Joint Training 
System, conducted 13 large-scale exercises since it was created in 2002, 
and generally completed exercise summary reports within the required 
time frame. However, those reports did not consistently include certain 
information; for example, only 5 of the 11 exercise summary reports 
included an identified section on lessons learned. DHS currently has a 
template for exercise documentation that includes guidance on content 
and format; however, NORTHCOM does not follow this template and 
neither DOD nor NORTHCOM has similar guidance that specifies content 
and format. Without a consistent record of what occurred during an 
exercise, NORTHCOM cannot ensure that it has met internal standards for 
planning and execution of joint exercises, compare exercise results over 
time, and share lessons learned with interagency partners and states. We 
are recommending that NORTHCOM's Commander establish criteria for 
the format and content of postexercise documentation. 

Interagency partners and states have participated in NORTHCOM 
exercises, but NORTHCOM faces challenges involving interagency 
partners and states in planning, conducting, and assessing exercises. 



Chairman of the Joint Chiefs of Staff, Joint Training Manual for the Armed Forces of the 
United States, 3500.03B (Washington, D.C.: Aug. 31, 2007), A-l; U.S. Northern Command, 
Operations: Exercise Program, Instruction 10-156 (Colorado Springs, Colo.: June 2, 2008). 
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NORTHCOM guidance requires that NORTHCOM maximize exercise 
participation with interagency partners and states to achieve preparedness 
goals and ensure a seamless operating environment. 20 Seventeen federal 
agencies and organizations and 17 states and the District of Columbia have 
participated in one or more of the seven large-scale exercises that 
NORTHCOM has conducted since September 2005. We found that the 
states we visited derived benefits from their participation in these 
exercises. For example, officials from 3 states told us this was their first 
opportunity to interact with federal military forces. However, we found 
that challenges remain which have resulted in inconsistencies in the way 
that NORTHCOM involves the states in its exercises. One of DOD's 
challenges is adapting its exercise system and practices to accommodate 
the coordination and involvement of other federal, state, local, and tribal 
agencies that do not have the same kinds of practices or level of planning 
effort. NORTHCOM also faces the challenge of balancing its training 
objectives with those of state agencies and organizations, particularly 
given the limited resources and funding states have available to exercise. 
While state and local governments seek to exercise their first responder 
capabilities before having their resources overwhelmed and needing to 
seek federal assistance, NORTHCOM's goal is to exercise its capability to 
provide support to civil authorities when local, state, and other federal 
resources are overwhelmed. As a result of this challenge, officials from 5 
states told us that all of their needs were not fully met during the 
exercises, for example, due to large-scale, unrealistic scenarios that 
overwhelmed the states' resources before they had the opportunity to 
exercise their training objectives. Inconsistencies with how NORTHCOM 
involves states in planning, conducting, and assessing exercises are 
occurring in part because NORTHCOM officials lack experience in dealing 
with the differing emergency management structures, capabilities, and 
needs of the states. Inconsistencies are also occurring because 
NORTHCOM has not established a process for including states in 
exercises, such as consistent procedures for requesting state involvement 
in exercises through DHS/FEMA or the National Guard Bureau. Without an 
informed and consistent process, NORTHCOM increases the risk that its 
exercises will not provide benefits for all participants, impacting the 
seamless exercise of all levels of government and potentially affecting 
NORTHCOM's ability to provide support to civil authorities. We are 
recommending that (1) DOD work with DHS to establish guidance and 
procedures for requesting state participation and involving states in 



20 U.S. Northern Command, Commander's Training Guidance FY 09-10, A-4. 
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planning, executing, and assessing exercises and (2) NORTHCOM's 
Commander develop a training plan for NORTHCOM staff on state 
emergency management structures and relevant issues related to working 
with civilian state and local emergency management officials. 

NORTHCOM has a systematic lessons learned and corrective action 
program to improve preparedness, but gaps remain with collecting and 
sharing lessons with agency and state partners and managing corrective 
actions. DOD and NORTHCOM guidance requires that NORTHCOM 
identify lessons learned during the course of normal operations, exercises, 
and real-world events; share valid observations and findings as widely as 
possible; and track them until the corrective actions are verified during 
subsequent events or exercises. 21 NORTHCOM uses DOD's Joint Lessons 
Learned Information System (JLLIS), an automated tool for managing and 
tracking observations, and a corrective action process to manage issues 
requiring action for resolution. Access to this system is limited for non- 
DOD participants, and DOD believes that DHS's system is not adequately 
protected from potential unauthorized users. NORTHCOM has taken steps 
to address these issues, such as placing a template on NORTHCOM's 
restricted access portal to provide a venue for non-DOD officials to record 
exercise observations, but these efforts have not been successful. Because 
NORTHCOM is not fully involving other federal agencies and states in its 
lessons learned process, it is missing opportunities to learn lessons from 
an exercise. For example, officials from two states did not provide 
NORTHCOM with observations from exercises because they did not 
attend the command's postexercise reviews for varying reasons, such as 
limited resources. As a result, NORTHCOM risks the reoccurrence of these 
issues. We also found that about 20 percent of the corrective actions 
tracked by NORTHCOM were being closed prematurely because, under 
existing procedures, NORTHCOM exercise directorate officials do not 
have oversight over issues that are resolved within other directorates or 
are not giving long-standing issues the sustained management attention to 
ensure resolution. A NORTHCOM official told us that they do not have the 
staff necessary to oversee the actions on records handled within the other 
directorates. Closing issues requiring corrective actions prematurely could 
increase the risk that they could reoccur and limits the knowledge gained 



Chairman of the Joint Chiefs of Staff, Joint Lessons Learned Program, Instruction 
3150.25D (Washington, D.C.: Oct. 10, 2008), A-l and A-6; U.S. Northern Command, 
Commander's Training Guidance FY 09-10 , A-4, A-8, A-20, A-2-8; U.S. Northern 
Command, Lessons Learned Program and Corrective Action Board Process, Instruction 
16-166 (Colorado Springs, Colo.: Mar. 16, 2009), 2. 
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and value of the exercise. We are recommending that (1) NORTHCOM's 
Commander establish and publicize procedures for non-DOD exercise 
participants to submit observations, (2) the Chairman of the Joint Chiefs 
of Staff direct the Joint Staff to work with DHS to either resolve 
information assurance issues so that combatant commands can post 
Exercise Summary Reports with lessons learned on DHS's Lessons 
Learned Information Sharing system or establish an alternative method to 
systematically collect and share lessons learned, and (3) the Chairman of 
the Joints Chiefs of Staff direct the Joint Staff to revise the joint lessons 
learned operating instruction to include procedures to ensure that 
corrective actions are implemented and verified in a subsequent exercise 
or operation before being closed. 

NORTHCOM has taken steps to integrate its exercises with the NEP, such 
as participating in several NEP exercises, including combining two of its 
large-scale exercises with NEP exercises. However, NORTHCOM has not 
consistently applied NEP planning and documentation requirements, 
because NEP guidance is unclear on the extent to which federal agencies 
should follow NEP requirements and DOD guidance does not specifically 
address this issue. NEP guidance provides policies and tools for the 
design, planning, conduct, and evaluation of exercises — known as the 
Homeland Security Exercise and Evaluation Program. 22 FEMA requires 
that entities, such as states, receiving homeland security grant funding for 
their exercises use Homeland Security Exercise and Evaluation Program 
tools, such as reporting templates. We reviewed NEP guidance, such as the 
Implementation Plan, and found it does not clearly state the extent to 
which federal agencies are required to follow the Homeland Security 
Exercise and Evaluation Program planning and documentation guidance. 
As a result of this unclear guidance, we found that agency officials have 
varying interpretations of the requirements. A DOD and a Joint Staff 
official told us that NEP guidance does not require agencies to use the 
Homeland Security Exercise Evaluation Program even for NEP exercises; 
therefore, NORTHCOM uses the Joint Training System as the basis for 
planning, conducting, and assessing exercises. FEMA officials told us that 
federal agencies should use the Homeland Security Exercise Evaluation 
Program for exercises that require some level of interagency participation, 



DHS, National Exercise Program Implementation Plan (Washington, D.C.: June 20, 
2008). 
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so that consistent approaches are used by the various participants. 23 
Neither DOD nor NORTHCOM guidance specifically addresses this issue. 24 
We recognize that DOD and NORTHCOM must meet their own mission 
and exercise requirements and the Joint Training System may be best 
suited for NORTHCOM's exercises; however, all of the states we visited 
use Homeland Security Exercise and Evaluation Program guidance. We 
found that having differing sets of guidance makes exercise development 
more difficult and potentially limits the benefits for participating states. 
Further, inconsistent approaches to the development and content of 
postexercise documentation may affect the ability of organizations to fully 
learn lessons identified in exercises. 25 We believe that achieving national 
preparedness requires a whole-of-government approach and is a shared 
responsibility among federal, state, and local governments and 
organizations and an integration of their various standards, policies, and 
procedures into the national system. 26 In the absence of clear NEP 
guidance on this issue, DOD should ensure that its components clearly 
understand when use of NEP planning and reporting guidelines is 
appropriate. We are recommending that the Secretary of Defense and the 
Joint Staff provide specific guidance that defines the conditions under 
which the combatant commands should follow the Homeland Security 
Exercise and Evaluation Program planning and documentation 
requirements or the DOD's Joint Training System should be modified for 
those civil support exercises. 

DOD provided written comments on a draft of this report and provided 
technical comments which we incorporated into the final report as 
appropriate. DOD agreed or partially agreed with all our recommendations 
and described actions it is taking or plans to take to implement them. DOD 



FEMA officials stated that federal departments and agencies should be held accountable 
for meeting key Homeland Security Exercise Evaluation Program requirements, but that 
FEMA lacks the authority to ensure compliance. While we recognize that FEMA's role is 
generally to coordinate, guide, and support, we believe that FEMA's expanded leadership 
role under the Post-Katrina Act of 2006 provides FEMA opportunities to instill a shared 
sense of responsibility and accountability on the part of all agencies. GAO-09-369. The Post- 
Katrina Act was enacted as Title VI of the Department of Homeland Security 
Appropriations Act, 2007, Pub. L. No. 109-295, (2006). 

24 NORTHCOM guidance states the Vigilant Guard Program should use Homeland Security 
Exercise and Evaluation Program formats to encourage civilian agency participation. U.S. 
Northern Command, Commander's Training Guidance: Fiscal Year 2009-2010, p. A-18. 

25 Defense Science Board, Unconventional Operational Concepts and the Homeland. 

26 GAO-09-369. 
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generally agreed with the recommendation to work with DHS to establish 
guidance and procedures for requesting state participation and involving 
states in planning, executing, and assessing exercises. DOD agreed that 
better coordination for interfacing with state officials can be achieved. 
However, DOD noted that NORTHCOM is the primary training audience 
for its exercises and that the recommendation should also be addressed to 
an interagency exercise planning committee. We noted NORTHCOM's 
considerable efforts to involve state and interagency exercise partners. 
However, we believe that the effective involvement of and interaction with 
state and other federal partners is a critical component of improving and 
maintaining NORTHCOM's preparedness for providing support to civil 
authorities regardless of whether it is a strictly NORTHCOM-sponsored 
exercise or an event conducted under the National Exercise Program. 
Therefore, developing procedures to improve coordination with the states 
can only assist DOD and all its interagency and state partners. We also 
noted that the interagency exercise planning committee is one venue at 
which DOD can effectively coordinate with its interagency partners. 
Although DOD agreed with the recommendation on a training plan for 
NORTHCOM staff and discussed current and future NORTHCOM training 
efforts, its response did not address the need for NORTHCOM staff to be 
provided more thorough training on specific emergency management 
organizations and structures of each state. We continue to believe that 
such training is needed since it is an important element of preparedness 
for exercises and for actually providing civil support in the event of a 
major incident. A summary of DOD's comments and a summary of our 
response to these comments follow the Recommendations for Executive 
Action section of this report. DOD's written comments are reprinted in 
appendix III. DHS also reviewed a draft of this report and provided 
technical comments, which we incorporated into the final report as 
appropriate. 



The U.S. homeland continues to face an uncertain, complex security 
environment with the potential for terrorist incidents and natural disasters 
which can produce devastating consequences. Ensuring an effective 
response will require that federal departments and agencies, states, and 
local governments conduct integrated disaster response planning and test 
these plans by exercising together. 



Training and Exercises Exercises play an instrumental role in preparing the nation to respond to 

an incident by providing opportunities to test emergency response plans, 
evaluate response capabilities, assess the clarity of established roles and 



Background 
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responsibilities, and improve proficiency in a simulated, risk-free 
environment. 27 Short of performance in actual operations, exercises 
provide the best means to assess the effectiveness of organizations in 
achieving mission preparedness. Exercises provide an ideal opportunity to 
collect, develop, implement, and disseminate lessons learned and to verify 
corrective action taken to resolve previously identified issues. 28 Sharing 
positive experiences reinforces positive behaviors, doctrine, and tactics, 
techniques, and procedures, while disseminating negative experiences 
highlights potential challenges in unique situations or environments or 
identifies issues that need to be resolved. 29 According to the National 
Response Framework, well-designed exercises improve interagency 
coordination and communications, highlight capability gaps, and identify 
opportunities for improvement. 30 There are various types of exercises 
ranging from tabletop exercises that involve key personnel discussing 
simulated scenarios in informal settings to a full-scale exercise, including 
many agencies, jurisdictions, and disciplines and a "boots on the ground" 
response, such as firefighters decontaminating mock victims. 



Military Mission and DOD established the Office of the Assistant Secretary of Defense for 

Organization Homeland Defense and Americas' Security Affairs to oversee homeland 

defense activities for DOD, under the authority of the Under Secretary of 
Defense for Policy, and, as appropriate, in coordination with the Chairman 
of the Joint Chiefs of Staff. This office develops policies, conducts 
analysis, provides advice, and makes recommendations on homeland 
defense, defense support of civil authorities, emergency preparedness, and 
domestic crises management matters within the department. The assistant 
secretary assists the Secretary of Defense in providing policy directions to 
NORTHCOM and other applicable combatant commands to guide the 



A training event focuses primarily on improving individual or collective ability to perform, 
such as academic or field training. An exercise focuses primarily on evaluating capability 
or an element of capability, such as a plan or policy. Necessary training should take place 
prior to an exercise. 

28 U.S. Northern Command, Commander's Training Guidance for FY09- 10, A-20. 

29 U.S. Northern Command, Lessons Learned Program and Corrective Action Board 
Process, Instruction 16-166 (Colorado Springs, Colo.: Mar. 16, 2009), 2. 

30 The framework is a guide to how the nation conducts all-hazards response and is 
intended to capture specific authorities and best practices for managing incidents that 
range from the serious but purely local, to large-scale terrorist attacks or catastrophic 
natural disasters. DHS, National Response Framework (Washington, D.C.: January 2008). 
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development and execution of homeland defense plans and activities. This 
direction is provided through the Chairman of the Joint Chiefs of Staff. 

The Chairman of the Joint Chiefs of Staff, as principal military advisor to 
the President and Secretary of Defense, has numerous responsibilities 
relating to homeland defense and civil support, including providing advice 
on operational policies, responsibilities, and programs. Furthermore, the 
Chairman of the Joint Chiefs of Staff and the Joint Staff are responsible for 
formulating joint training policy and doctrine. 31 The Joint Staff assists the 
Chairman by facilitating implementation of the Chairman's joint training 
programs, including the Joint Training System, Chairman's sponsored 
exercise program, and joint exercise program. 



NORTHCOM NORTHCOM is the military command responsible for planning, organizing, 

and executing DOD's homeland defense and civil support missions within 
its area of responsibility — the continental United States (including Alaska) 
and territorial waters (see fig. I). 32 Homeland defense is the protection of 
U.S territory, sovereignty, domestic population, and critical defense 
infrastructure against external threats and aggression. DOD is the primary 
federal agency responsible for homeland defense operations, such as air 
defense, and NORTHCOM is the combatant command responsible for 
commanding and coordinating a response to a homeland defense 
incident. 33 To carry out its homeland defense mission, NORTHCOM is to 
conduct operations to deter, prevent, and defeat threats and aggression 
aimed at the United States. 



Chairman of the Joint Chiefs of Staff, Instruction 3500. 01E, Joint Training Policy and 
Guidance for the Armed Forces of the United States (Washington, D.C.: May 31, 2008). 

32 DOD, Strategy for Homeland Defense and Civil Support, 8. 

33 Homeland defense is considered DOD's portion of the broader area of homeland security. 
DHS is the lead primary federal agency for homeland security issues, which is a concerted 
national effort to prevent terrorist attacks within the United States, reduce America's 
vulnerability to terrorism, and minimize the damage and recover from attacks that do 
occur. Homeland Security Council, National Strategy for Homeland Security, 3; DOD, 
Strategy for Homeland Defense and Civil Support, 5. 
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Figure 1 : NORTHCOM s Area of Responsibility 




Sources: GAO analysis; Map Resources (map). 



NORTHCOM's second mission is civil support or defense support of civil 
authorities. Civil support is DOD support to U.S. civilian authorities, such 
as DHS, for domestic emergencies, both natural and man-made, and 
includes the use of DOD personnel — federal military forces and DOD's 
career civilian and contractor personnel — and DOD agency and 
component resources. Because these missions are complex and 
interrelated, they require significant interagency coordination. Civil 
support missions include domestic disaster relief operations for incidents 
such as fires, hurricanes, floods, and earthquakes. Such support also 
includes counterdrug operations and management of the consequences of 
a terrorist incident employing a weapon of mass destruction. DOD is not 
the primary federal agency for such missions (unless so designated by the 
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President) and thus provides defense support of civil authorities only 
when (1) state, local, and other federal resources are overwhelmed or 
unique military capabilities are required; (2) assistance is requested by the 
primary federal agency; and (3) NORTHCOM is directed to do so by the 
President or the Secretary of Defense. 34 See fig. 2 for the pathway for 
requesting DOD and NORTHCOM assistance during an incident. 



DHS, National Response Framework; Joint Chiefs of Staff, Joint Pub. 3-28, Civil Support 
(Washington, D.C.: Sept. 14, 2007). 
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Figure 2: Pathways for Requesting Military Assistance for Incident Response 
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Source: GAO analysis of DOD and DHS documentation. 
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NORTHCOM conducts or participates in exercises to improve readiness to 
perform its assigned missions. The command annually conducts 2 large- 
scale exercises 35 — Ardent Sentry and Vigilant Shield — and participates in 
over 30 smaller command, regional, state, and local exercises. Each Ardent 
Sentry and Vigilant Shield training event emphasizes one of the key 
missions while at the same time including elements of the other. Ardent 
Sentry emphasizes the civil support missions; Vigilant Shield the homeland 
defense missions. The basis for NORTHCOM's exercises is DOD's Joint 
Training System. NORTHCOM's Training and Exercise Directorate is 
responsible for planning and executing joint training, exercises, and 
education programs to ensure NORTHCOM is prepared to accomplish its 
assigned missions. 



DOD's Joint Training Due to the need to prepare for and conduct military operations to defend 

System t ne United States and fight the nation's wars, DOD has developed an 

established, authoritative, time-tested process for planning, conducting, 
and evaluating exercises in order to test and improve preparedness to 
meet its wide range of critical missions. 36 NORTHCOM uses DOD's Joint 
Training System as the basis to design, develop, and conduct exercises. 37 
The Joint Training System provides an integrated, requirements-based 
method for aligning training programs with assigned missions consistent 
with command priorities, capabilities, and available resources. The joint 
system consists of four phases beginning with the identification of critical 
capabilities required based on assigned missions, proceeding through the 
planning and scheduling of training events, the execution and evaluation 
of required training, and assessing training proficiency against required 
capability (see fig. 3). This process is designed to ensure that an 
organization's training program is linked to the Joint Mission Essential 



°For purposes of this report, large-scale exercises are those involving multiple agencies 
and organizations, venues, and events, such as NORTHCOM's Vigilant Shield and Ardent 
Sentry exercises. 

36 DOD established the Joint Training System in fiscal year 1994. GAO, Military Training: 
Actions Needed to Enhance DOD's Program to Transform Joint Training, GAO-05-548 
(Washington, D.C.: June 21, 2005) 3, 19. 

7 The documents that provide the written framework for the Joint Training System are 
Chairman of the Joint Chiefs of Staff, Instruction 3500.01E, Joint Training Policy and 
Guidance for the Armed Forces of the United States (Washington, D.C.: May 31, 2008); 
Manual 3500.03B, Joint Training Manual for the Armed Forces of the United States 
(Washington, D.C.: Aug. 31, 2007); and Manual 3500.04E, Universal Joint Task Manual 
(Washington, D.C.: Aug. 25, 2008). 
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Task List, 38 the personnel executing the tasks are properly trained, and 
shortfalls in training are identified and corrected in order to improve 
readiness. The Joint Training Information Management System is an 
automated system specifically designed to assist users in managing 
elements of each of the four phases of the Joint Training System. 39 During 
the execution phase, commanders and directors focus on executing and 
evaluating planned training events, which can be accomplished through 
academic training, exercises, or a combination of these activities. 40 During 
the execution stage of the Joint Training System, the Joint Event Life 
Cycle provides a five-stage methodology for joint-event development 
design, planning, preparation, execution, and evaluation. For example, 
DOD components prepare for the execution of an exercise by conducting 
five conferences, such as the Concept Development Conference where 
exercise and training objectives are discussed and scenarios developed. 
Activities for the Joint Event Life Cycle are managed through the Joint 
Training Information Management System. 



The Joint Mission Essential Task List outlines those tasks that are essential to a 
combatant command's ability to perform assigned missions. The list is a key element in 
readiness assessment and reporting. 

39 Specifically, the Joint Training Information System provides the joint community with an 
integrated software capability to identify, collect, analyze, store, and disseminate the data 
required to execute the Joint Training System. 

40 Joint exercises have been characterized by some form of multiechelon, computer-assisted 
exercises which embody complex simulation. These may be either a field training exercise, 
command postexercise, or a tabletop exercise. 
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Figure 3: DOD's Joint Training System 
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Evaluating lessons learned and identifying issues for corrective actions are 
fundamental components of DOD's training and exercise process. The 
Chairman of the Joint Chiefs of Staff provides policy, direction, and 
guidance for DOD's Joint Lessons Learned Program. 41 The objectives of 
this program are to collect and analyze observations from exercises and 
real world events; disseminate validated observations and findings to 
appropriate officials; identify and implement corrective actions; and track 
corrective actions until reobserved in a subsequent exercise or event to 



Chairman of the Joint Chiefs of Staff, Join t Lessons Learned Program, 3150.25D, C-l. 
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ensure that the issue has been successfully resolved. Combatant 
commands, including NORTHCOM, execute lessons discovery, knowledge 
development, and implementation activities scaled to meet the command's 
requirements while supporting and feeding into the Chairman's Joint 
Lessons Learned Program by identifying lessons applicable across 
combatant commands and the services. 



The National Exercise The NEP was established in April 2007 under the leadership of the 

Program Secretary of Homeland Security to prioritize and coordinate federal, state, 

and local exercise activities and serves as the principal mechanism for 
examining the preparation of the federal government to respond to an 
incident and adopting policy changes to improve such preparation. 42 The 
day-to-day staff-level coordination of the NEP is managed by the NEP 
Executive Steering Committee — a working group of the White House's 
Domestic Readiness Group Exercise and Evaluation Sub-Policy 
Coordination Committee — and is chaired and facilitated by FEMA's 
National Exercise Division. The steering committee is also responsible for 
framing issues and recommendations for the full coordination committee 
on exercise themes, goals, objectives, scheduling, and corrective actions. 43 
Figure 4 illustrates the major events and milestones of the NEP and 
NORTHCOM's exercise program and table 1 provides information on 
related major documents. 



DHS, National Exercise Program Implementation Plan. 

43 Other steering committee members include DOD, the Office of the Director of National 
Intelligence, the Federal Bureau of Investigation, and the Departments of State, Justice, 
Energy, Transportation, and Health and Human Services. 
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Figure 4: National Exercise Program and U.S. Northern Command Exercise Program Time line: Major Events and Milestones 
(2001-2009) 
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Source: GAO Analysis of DOD and DHS documentation. 
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Table 1 : Major Documents Related to the National Exercise Program and NORTHCOM 



Title 



Description 



Homeland Security Presidential Directive-8 



Requires the development of the national preparedness goal; preparedness 
priorities; an assessment system; a training and exercise program; a system to 
collect, analyze, and disseminate information from exercises, training events, and 
actual incidents; equipment standards; and a federal response capability 
inventory. 



National Response Framework 



Establishes a comprehensive, national, all-hazards approach to domestic incident 
response. Replaced the National Response Plan. 



National Preparedness Guidelines 



Establishes a vision for national preparedness and provide a systematic approach 
for prioritizing preparedness efforts across the United States 



National Planning Scenarios 



Provides planning tools that represent a minimum number of credible scenarios 
depicting the range of potential terrorist attacks and natural disasters and related 
impacts facing the United States They form a basis for coordinated federal 
planning, training, and exercises. 



U.S. North American Aerospace Defense 
Command (NORAD)-NORTHCOM Instruction 
10-156 



Establishes the NORAD and NORTHCOM exercise program, provides an 
overview of the exercise program, describes the principal NORAD and 
NORTHCOM-sponsored exercises, and outlines the process for scheduling, 
planning, execution, and afteraction review of exercises. 



Chairman of the Joint Chiefs of Staff Instruction 
3150.25D 



Codifies the Joint Lessons Learned Information System as the DOD system of 
record for the Joint Lessons Learned Program. 



DOD Instruction 3020.47 



Establishes policy, assigns responsibilities, and provides instructions for DOD 
participation in the National Exercise Program. 



Source: DOD and DHS. 



The NEP includes a series of national exercises projected on a 5-year 
exercise schedule. These exercises are organized into four tiers with each 
tier reflecting different requirements for interagency participation (see fig. 
5). 
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Figure 5: Description of NEP Tiers 



White House directed, U.S. Government-wide 
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Other federal exercises, regional operational, tactical or 
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conducted annually depends upon individual entities' 
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Non-Federal Exercises, State, Territorial, Local and or 
tribal governments and/or private sector focus. Federal 
Agency participation is optional. The number of 
exercises conducted annually depends upon individual 
entities' needs and requirements. 



Source: FEMA. 

FEMA administers the NEP and maintains the Homeland Security 
Exercise and Evaluation Program — a capabilities and performance-based 
exercise program — to provide standardized policy, methodology, and 
terminology for exercise design, development, conduct, evaluation, and 
improvement planning. DHS maintains policy and guidance for this 
program. 44 

Similar to DOD's Joint Training System, the Homeland Security Exercise 
Evaluation Program uses an exercise life cycle with five phases: 
foundation, design and development, conduct, evaluation, and 
improvement planning. This program also provides document templates 
for exercise planning and evaluation and a collection of interactive, on-line 
systems for exercise scheduling, design, development, conduct, 
evaluation, and improvement planning, referred to as the Homeland 






DHS, Homeland Security Exercise and Evaluation Program, Volumes 1-5 (Washington, 
D.C.: February 2007). 
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Security Exercise Evaluation Program Tool Kit (see fig. 6). 45 FEMA also 
has additional resources to support exercises. For example, exercise 
stakeholders can access FEMA's Lessons Learned Information Sharing 
system, an interagency Web site for posting lessons learned and sharing 
best practices, to learn about promising practices that could facilitate 
exercise activities. 46 



45 https://hseep. dhs.gov/pages/1001_Toolk.aspx. 
46 https://llis.gov. 
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Figure 6: Homeland Security Exercise and Evaluation Program Tool Kit 
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National Exercise Schedule System 

A national online comprehensive tool that facilitates scheduling and 
synchronization of national-level, federal, state, and local exercises. 



Design, development, and conduct 



Design and Development System 

A project management tool and comprehensive tutorial for the design, 
development, conduct, and evaluation of exercises. 



Exercise Evaluation Guide Builder (Beta) 

An online application that enables users to customize exercise evaluation 

guides and templates. 



Master Scenario Events List Builder (Beta) 

A tool that enables users to create customized master scenario events list 
formats by selecting from a list of data fields. 



Evaluation and Improvement Planning 



Corrective Action Program System 

An online application that enables users to prioritize, track, and analyze 
improvement plans developed from exercises and real-world events. 



Source: GAO analysis of FEMA documentation. 
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NORTHCOM's 
Exercise Program Is 
Consistent with 
DOD's Joint Training 
System, but Exercise 
Reporting is 
Inconsistent 



NORTHCOM's Commander's Training Guidance requires that NORTHCOM 
establish a training and exercise program consistent with the Joint 
Training System and establishes that training efforts and resources will be 
focused on two large-scale exercises annually. 47 The Joint Training System 
requires, among other things, that an organization's training objectives be 
linked to its Joint Mission Essential Task List and include the use of the 
Joint Events Life Cycle for planning, conducting, and assessing exercises. 48 
We found that NORTHCOM has developed a comprehensive exercise 
program consistent with DOD's Joint Training System. For example, 
NORTHCOM uses the Joint Training Information Management System to 
link training objectives with its Joint Mission Essential Task List. 
NORTHCOM officials enter information on task performance of exercise 
participants into the Joint Training Information Management System to 
evaluate the extent to which the command is trained based on 
performance requirements in the Joint Mission Essential Task List. 49 
NORTHCOM also uses the Joint Training Information Management System 
to manage the Joint Events Life Cycle for its large-scale exercises, 
including planning exercise milestones and developing a time line that 



U.S. Northern Command, Commander's Training Guidance for FY 09-10, A-ll. 

48 The Joint Events Life Cycle provides a sequence of the inputs, process steps, and outputs 
necessary to execute any size exercise. This process consists of five stages including 
design, planning, preparation, execution, and evaluation. A series of planning conferences 
provide discrete breakpoints between each stage. Chairman of the Joint Chiefs of Staff, 
Joint Training Manual for the Armed forces of the United States, E-6. 

49 NORTHCOM is given an overall rating in the system for the command's current status of 
"trained", "needs practice," or "untrained" for each Joint Mission Essential Task. U.S. 
Northern Command, Memorandum on NORAD and USNORTHCOM Joint Training Plan 
FFO5-09(Colorado Springs, Colo.: July 30, 2007, A-32-A-35; Chairman of the Joint Chiefs of 
Staff, Joint Training Manual for the Armed Forces of the United States, F-5-F-6. 



NORTHCOM's Exercise 
Program is Consistent with 
DOD's Joint Training 
System 
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allows exercise planners to see where they are in the event life-cycle 
process. For example, NORTHCOM holds five planning conferences for 
each exercise, including a concept development conference, where 
exercise and training objectives are discussed and scenarios developed. 50 
We also found that NORTHCOM has conducted 13 large-scale exercises 
since it was created in 2002, generally including 2 exercises each year (see 
table 2). Vigilant Shield is held in the fall and focuses primarily on 
NORTHCOM's homeland defense mission, and Ardent Sentry is generally 
conducted in the spring and focuses on defense support of civil 
authorities. 



Table 2: Large-Scale NORTHCOM Exercises Since 2002 



Exercise 


Dates 


Scenarios 


Ardent Sentry 09 


June 18-24, 2009 


Flooding along Mississippi River 

Domestic terrorist organization releases foot and mouth disease at 4-H expo 
Anthrax released into the food supply 

Train derailments in Iowa and Kansas resulting in chemical releases 

Domestic terrorist organization launches attack against a Wyoming Air Force 
Base 

U.S. and Canadian maritime incidents 


Vigilant Shield 09 


Nov. 12-18, 2008 


Earthquake along San Andreas Fault 

Homeland defense vignettes associated with mission areas 


Ardent Sentry 08 


May 1-8, 2008 


Category 4 hurricane strike and terrorist threat affecting National Capital 
Region 

Accidental VX nerve gas release in Oregon 

Multiple terrorist attacks in Washington to exercise both response to chemical 
weapons attack and specific defense support of civil authorities capabilities 


Vigilant Shield 08 


Oct. 15-20, 2007 


Series of exploded radiological dispersal devices 
Aircraft accidents 
Reverse no-fly list 

Deployment of a command assessment element 


Ardent Sentry 07 


April 30 - May 17,2007 


Critical infrastructure protection 
Maritime operations 

Detonation of a 10-kiloton improvised nuclear device in Indiana 
Category III hurricane impacting the New England area 



Chairman of the Joint Chiefs of Staff, Joint Training Manual for the Armed Forces of the 
United States, E-6-E-12. 
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Exercise Dates Scenarios 



Vigilant Shield 07 


Oct. 16-27, 2006 and 


Crash and explosion of plane carrying nuclear weapons 




Dec. 4-15, 2006 


Maritime attacks/events 

Rogue aircraft and North American Air Defense Command Air and Strategic 
Threat events 

Strategic Ballistic Missile Defense Force Protection 
Defense against cyber attacks 


Ardent Sentry 06 


May 8-18, 2006 


Hurricane strike on New Orleans, Louisiana 
Explosion of a railcar carrying phosgene 
• U.S. and Canadian Infrastructure attack 
Chemical attack 
Radiological incident 
Avian flu pandemic outbreak scenario 


Vigilant Shield 06 


Nov. 1-10, 2005 


Chemical scenario 

Critical infrastructure protection of a nuclear power plant 

Biological scenario 

Ballistic missile defense threat 

Aerospace defense 


Ardent Sentry 05 


Apr. 4-9, 2005 


Exercise deployment and employment of Joint Task Force-Civil Support in 
support of FEMA Region I mission assignment requests in a chemical 
response 

Response to a biological event 

Protection of critical infrastructure (nuclear power plant) 


Determined Promise 04 


Aug. 5-10, 2004 


Simultaneous terrorist air events (hijacked aircraft, aircraft with terrorists on 
watch list, low-altitude defense and swarm attack by business jet aircraft) 

Explosion of container with radiological dispersal device in Long Beach, 
California harbor 

• Train derailment and chemical spill 

Multiple sarin, mustard, and chlorine gas attacks in Virginia 

Maritime intercept operations 

Attack by unmanned aerial vehicles with high-explosive warheads fired from 
oil platform 

Cyber security event 

Ballistic missile defense 


Unified Defense 04 


Feb. 19-25, 2004 


Air threats to Alaska, Canada, and the United States 
Category IV+ hurricane in Corpus Christi, Texas 
Nuclear detonation in Texas 

Quick reaction force deployments to Texas and Alaska 
Transportation security and U.S. Coast Guard incidents in Alaska 
Maritime intercept operations events 
Ballistic missile defense threat 



Page 28 



GAO-09-849 Homeland Defense 



Exercise 


Dates 


Scenarios 


Determined Promise 03 


Aug. 18-28, 2003 


Terrorist attack of a biological weapon of mass destruction in Clark County, 




Nevada 

Wildfires in the western United States 

2003 World Gymnastics Championship in California 

Ongoing operational plan load-out operations 

Sustained, diverse maritime events 

Air threat to Alaska 

High-interest vessel threat 

Strategic infrastructure threats in the northwest United States 
Category III hurricane in the southeast United States 
Train derailment in Kentucky 
Shipping and security issues in Alaska 


Unified Defense 03 


Feb. 6-13, 2003 a 


National special security event - Super Bowl 

J ointTask Force-Civil Support held in strategic reserve for State of the Union 
Address and Super Bowl 

Nuclear threat along southwest U.S. border 

Border surveillance 

Threat to port 

Consequence management in northeast United States 
Natural disaster in Alaska 



Source: GAO analysis of NORTHCOM documents. 



"Unified Defense 03 started late due to the Columbia Shuttle disaster and ended early due to a level 
Orange alert in the Homeland Security Alert System. 



NORTHCOM guidance outlines the postexercise documentation required 
to be completed for each exercise, including quick look, after-action, and 
exercise summary reports; provides a time line for the completion of these 
documents; and includes general direction that these documents follow 
the same focus areas as the collection management plan — the source 
document from which exercise analysts identify, examine, and 
recommend emerging issues and trends. 51 We found that NORTHCOM has 
generally completed exercise summary reports for its exercises; however, 
neither NORTHCOM nor Joint Forces Command officials could locate an 
exercise summary report for Unified Defense 03. 52 In addition, 



U.S. Northern Command, Operations: Exercise Program, Instruction 10-156 (Colorado 
Springs, Colo.: June 2008), 20. 

52 While NORTHCOM was unable to provide exercise summary or quick-look reports for 
United Defense 03, they did provide pre-exercise briefings. NORTHCOM officials told us 
that the exercise started late due to the Columbia shuttle disaster and ended early due to a 
level Orange alert in the Homeland Security Alert System. 



NORTHCOM's 
Documentation of 
Exercises Is Generally 
Timely but Inconsistent 
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postexercise documentation is not consistently included on NORTHCOM's 
portal or the Joint Training Information Management System. NORTHCOM 
guidance issued in June 2008 provides a time line for the completion of 
postexercise documents and has been applicable to 2 subsequent 
exercises-Ardent Sentry 08 and Vigilant Shield 09. According to the 2008 
guidance, the exercise summary report is to be submitted to the 
NORTHCOM Commander within 90 days of completing an exercise. 53 The 
Ardent Sentry 08 and Vigilant Shield 09 exercise summary reports were 
issued 99 days and 92 days, respectively, after the completion of each 
exercise. Overall, we reviewed exercise summary reports for 11 of 
NORTHCOM's large-scale exercises that have taken place since 2003. 
Seven of the 11 exercise summary reports were issued within 100 days. 
Four of the reports were issued later than 100 days, and 1 of 
NORTHCOM's earlier reports was issued in less than 30 days. 

NORTHCOM guidance states that exercise summary reports should 
provide the official description of the exercise, identify significant lessons 
learned, and be targeted toward a national audience. 54 Guidance also 
requires that exercise summary reports follow the same focus areas as the 
collection management plan — the source document from which exercise 
analysts identify, examine, and recommend emerging issues and trends. 55 
We found that NORTHCOM's exercise summary reports generally included 
an executive summary, training objectives, and the exercise's major 
scenarios and events, but did not consistently include lessons learned, 
exercise strengths and weaknesses, or clear recommendations. The 
exercise summary reports that included a section on lessons learned 
lacked details. For example, 6 of the 11 exercise summary reports we 
reviewed included an identified lessons learned section, and just 1 of these 
6 reports — Unified Defense 04 — provided additional information on 
lessons learned beyond identifying the title of each observation and the 
status of the observation in the lessons learned management system. As 



U.S. Northern Command, Operations: Exercise Program. The instruction, issued in June 
2008, also requires exercise quick look reports be issued within 30 days after the 
completion of an exercise. The instruction is only applicable to Ardent Sentry 08 and 
Vigilant Shield 09, as the post-exercise documents for these reports were issued after June 
2008. Post-exercise documents for all other exercises reviewed by GAO were issued prior 
to June 2008. 

54 U.S. Northern Command, Operations: Exercise Program, 20. 
55 U.S. Northern Command, Operations: Exercise Program, 20. 
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discussed later in this report, access to this system is required in order to 
obtain any additional information on the lesson learned. 56 

We also found that NORTHCOM exercise summary reports have not 
followed the same focus areas as collection management plans. 57 For 
example, none of the seven exercise summary reports for NORTHCOM 
exercises conducted since Hurricane Katrina in 2005 reported on the 
information identified in the collection management plans' focus areas. 
Inconsistencies in exercise documentation may be occurring because DOD 
and NORTHCOM guidance do not require a standard format or specific 
content for postexercise documentation. Although NORTHCOM uses other 
methods to document exercises, such as the Joint Training Information 
Management System, this system does not include a complete record of 
each exercise. For example, the Joint Training Information Management 
System does not include the lessons learned from an exercise. In addition, 
access to this system is generally limited to DOD officials. Recognizing the 
need for a complete and consistent record of each exercise, DHS's 
Homeland Security Exercise Evaluation Program provides a template for 
exercise documentation, including format and content. 58 NORTHCOM 
used this template for National Level Exercise 2-08, but does not use the 
template for its own exercises. 59 Despite differences in the requirements 
and complexities of NORTHCOM's and DHS' exercise programs, the lack 
of a complete and consistent record of each exercise lessens the extent to 
which NORTHCOM can ensure it has trained to key focus areas. Further, it 
deprives the command of a key source of historical information upon 
which to base current and future assessments of exercises and a 



Access to JLLIS is restricted in order to safeguard sensitive information because of 
concerns that adversaries may be able to exploit any weaknesses contained in those 
records. For users to access the unclassified portion of the JLLIS they must have a DOD 
Common Access Card, a card reader, and appropriate computer software prior to 
requesting access. 

57 Collection Management Plans are generally linked to the Joint Mission Essential Tasks 
and exercise objectives. Focus areas include a wide range of operations, such as 
coordinating and integrating regional interagency activities and coordinating consequence 
management in theater. U.S. Northern Command, Operations: Exercise Program, 20. 

58 DHS, Homeland Security Exercise Evaluation Program, Volume III: Exercise 
Evaluation and Implementation Plan, (Washington, DC: February 2007), A-l - B-2. 

59 U.S. Northern Command, North American Aerospace Defense Command and United 
States Northern Command National Level Exercise 2-08: 1-8 May 2008 Exercise 
Summary Report, (Colorado Springs, Colo.: Aug. 15, 2008)1-25. 
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consistent venue for sharing lessons learned with interagency partners and 
states. 



Interagency Partners 
and States Have 
Participated In 
NORTHCOM's 
Exercises, but 
Challenges Remain 
with Including States 
in Planning, 
Conducting, and 
Assessing Exercises 

NORTHCOM recognizes the importance of exercising with key partners in 
all its missions and that, in order to achieve its goal of being trained and 
ready to execute joint operations and ensure a seamless operating 
environment, NORTHCOM should maximize exercise participation with 
federal, state, and local agencies and National Guard units. 60 NORTHCOM 
has included interagency partners, such as DHS, FEMA, and the U.S. Coast 
Guard, and several states in its large-scale exercises (see table 3). We 
found that 17 civilian federal agencies and organizations have participated 
to varying degrees in one or more of the seven large-scale NORTHCOM 
exercises that have occurred since Hurricane Katrina made landfall in 
August 2005. Seventeen states have participated in NORTHCOM exercises 
since that time, and 8 of these states — Arizona, California, Connecticut, 
Indiana, Michigan, Oregon, Rhode Island, and Washington — played a 
major role by having a portion of the exercise conducted in the state and 



Federal Agencies and 
States Have Participated in 
NORTHCOM Exercises 



U.S. Northern Command, Commander's Training Guidance for FY09- 10; Chairman of 
the Joint Chiefs of Staff, Joint Training Manual for the Armed Forces of the United 
States, CJCSM 3500.03B (Aug. 31, 2007). 
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having various state agencies and officials participate. 61 For example, 
Indiana and Rhode Island played major roles in Ardent Sentry 07 for the 
detonation of a 10-kiloton improvised nuclear device and category III 
hurricane impacting the New England region, respectively. Both states 
established emergency operating centers and exercised large numbers of 
state emergency management personnel. 



Table 3: Interagency Partner and State Participation in Large-scale NORTHCOM Exercises Since 2006 



Exercise 


Other federal agencies 
and organizations 


States 


Vigilant Shield 06 


Department of Homeland Security 


Connecticut 




Other federal organizations in the areas of homeland defense 
and defense support of civil authorities" 


New Jersey 


Ardent Sentry 06 


Coast Guard Sector Detroit District 9 


Arizona 




Department of Energy 


- Division of Emergency 




Department of Health and Human Services 


Management 




Department of Homeland Security 


- Joint Terrorist Task Force 




Department of Justice 


- Tactical Intelligence Center 




Department of State 


Maine 




Department of Transportation 


Michigan 




Environmental Protection Agency 


- Detroit Metropolitan Airport 




Federal Aviation Administration 


- State Emergency Operation 




Federal Bureau of Investigation 
. FEMA 


Center 
- State Police 




Transportation Security Agency 
U.S. Coast Guard/Atlantic Area 
U.S. Department of Agriculture 


- Service Emergency 
Preparedness Liaison Officer 

- St. Clair County Regional 
Response Team 



- Wayne County 



We visited all of these states except for Connecticut and Indiana. We visited Indiana 
during a review of NORTHCOM's planning and interagency coordination in the spring of 
2007. We also met with Nevada officials who participated in Determined Promise 03. The 
remaining 9 of the 17 states played lesser roles in the exercises. 
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Other federal agencies 




Exercise 


and organizations 


States 


Vigilant Shield 07 


Department of Energy 


Arizona 




DHS National Interagency Simulation Cell 


- Pima County Emergency 




Department of Justice 


Management 




Department of Transportation 


- Tucson Police and Fire 




. FEMA 


• Washington, UO 




DHS Principal Federal Official 


Alaska 




U.S. Coast Guard 






Department of Agriculture 






Environmental Protection Agency 






Federal Bureau of Investigation 






Department of Health and Human Services 






DHS Immigration and Customs Enforcement 






National Geospatial Intelligence Agency 






DHS Transportation Security Administration 






National Security Agency 






Veterans Administration 




Ardent Sentry 07 


DHS-National Operations Center 


Indiana 




. FEMA 


- Department of Homeland Security 




FEMA Region 1 


- Indianapolis, Indiana 




FEMA Region II 


- Marion County, Indiana 




FEMA Region X 


Rhode Island 




Federal Bureau of Investigation 


Connecticut 




U.S. Coast Guard Pacific Area 


Massachusetts 



Vermont 
New Hampshire 
Maine 
New York 
Ohio 
Illinois 
Alaska 
Oregon 

- City of Portland 
Arizona 

- City of Phoenix 
Guam 3 

Department of Homeland Security • Washington, DC 

FEMA National Exercise Division • Oregon 

Other federal organizations in the areas of homeland defense • Washington 
and defense support of civil authorities" 



National Level 
Exercise 1-08/Vigilant 
Shield 08 



FEMA 

DHS-National Operations Center 
FEMA Regions IX and X 
U.S. Coast Guard Pacific Area 



National Level 
Exercise 2-08/Ardent 
Sentry 08 
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Exercise 


Other federal agencies 
and organizations 


States 


Vigilant Shield 09 


. FEMA Region IX 

Other federal organizations in the areas of homeland defense 
and defense support of civil authorities" 


California 


Ardent Sentry 09 


Dept of Energy 


Iowa 




Dept of Health and Human Services 

Dept of Homeland Security 

Environmental Protection Agency 

Federal Aviation Administration 

Federal Bureau of Investigation 
. FEMA Region VII 
. FEMA Region VIII 

U.S. Coast Guard Operational Command 

FEMA National Response Coordination Center 


- Dept of Agriculture 

- Dept of Health 

- Dept of Public Safety 

- Homeland Security & Emergency 
Management Division 




- Iowa State Emergency 




Operations Center 

- City of Des Moines Fire Dept. 

- City of Des Moines Police Dept. 



Kansas 

- Dept of Emergency Management 

- Dept of Health/Environment 

- Dept of Agriculture 

- Dept of Animal Health 

- Dept of Transportation 

- Highway Patrol 
Nebraska 
Missouri 
Wyoming 

Source: GAO analysis of NORTHCOM exercise documents. 

The US territory of Guam is in U.S. Pacific Command's area of responsibility. 

"Exercise summary reports for Ardent Sentry 08 and Vigilant Shield 06 and 09 indicate that other 
federal organizations in the areas of homeland defense and defense support of civil authorities 
participated in the exercises, but the reports do not provide specific names of organizations and we 
were unable to identify these organizations in the Joint Training Information Management System. 

State emergency management and National Guard officials told us that 
they participated in NORTHCOM exercises because they wanted to better 
understand the (1) capabilities that NORTHCOM could bring to the 
response to an incident and (2) command and control issues of the troops 
in a state when NORTHCOM is involved. We previously reported that 
states' participation in NORTHCOM exercises helps to build relationships 
and improve coordination. 62 Officials from all of the states we met with 



GAO-08-252. 
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told us that they derived benefits from their participation in these 
exercises. For example, state emergency management officials from three 
states told us that first-hand interaction with federal military forces and 
the opportunity to observe the federal response to an incident was 
beneficial. In addition, two state emergency management and National 
Guard officials told us that NORTHCOM officials were professional, well- 
trained, and helpful. Further, officials from five states told us that 
NORTHCOM provided beneficial resources, such as funds for travel to 
attend exercise planning conferences and contractor staff to help state 
officials prepare exercise scripts and injects. 63 Finally, officials from two 
states told us that the benefits of working with NORTHCOM included 
gaining an understanding of the resources and capabilities that 
NORTHCOM can provide, as well as understanding how NORTHCOM 
coordinates its response through FEMA. 

NORTHCOM is also attempting to include states in exercises through the 
Vigilant Guard Program. The goal of the Vigilant Guard Program is to 
enhance National Guard and State emergency management agency 
preparedness to perform their homeland defense and Defense Support to 
Civil Authorities roles and responsibilities. It focuses on State Guard Joint 
Force Headquarters coordination with the state emergency management 
agency and Joint Task Force-State operations and involves multiple states 
and agencies. The program began in September 2004 and included one 
exercise in fiscal year 2005. Now the plan is to conduct four exercises 
annually. NORTHCOM was given management responsibility for the 
Vigilant Guard exercises in 2007, although the National Guard Bureau 
retains responsibility for budgeting for these events. Two of the four 
annual Vigilant Guard exercises are to be linked to major combatant 
command exercises, usually NORTHCOM's Ardent Sentry and Vigilant 
Shield. States hosting a Vigilant Guard exercise determine the objectives 
for these events, and NORTHCOM provides support. Separate planning 
begins for these Vigilant Guard exercises prior to the related planning 
meetings for any linked NORTHCOM exercise. NORTHCOM's Ardent 
Sentry 09 is linked with a Vigilant Guard exercise in Iowa with scenarios 
including a train derailment and a chemical spill, an epidemic outbreak, 
and terrorism incident. 



Injects are pieces of information presented to the training audience during an exercise in 
a manner similar to real-world operations to guide the training audience toward the 
accomplishment of the training objectives. 
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NORTHCOM Faces 
Challenges Involving 
Interagency Partners and 
States in Planning, 
Conducting, and Assessing 
Exercises 



A key element to developing effective working relationships with all states 
is a well-thought out and consistent process for including the states in 
planning, conducting, and assessing exercises. Without such a process, 
states may be unwilling to participate in future NORTHCOM exercises, 
impacting the seamless exercise of all levels of government and potentially 
affecting NORTHCOM's ability to provide support to civil authorities. We 
found that challenges remain which have resulted in inconsistencies in the 
way that NORTHCOM involves the states in its exercises. One of DOD's 
challenges is adapting its exercise system and practices to accommodate 
the coordination and involvement of other federal, state, local, and tribal 
agencies that do not have the same kinds of practices or level of planning 
effort. Differences in exercise culture stem from differences in missions, 
experience, authority, scope, and resources available to DOD, interagency 
partners, and states. DOD has an established, authoritative, time-tested 
process for planning, conducting, and evaluating exercises in order to test 
and improve preparedness to meet its wide range of critical missions. 
Within DOD, training and exercises are considered a vital component of its 
overall mission of defending the national interests and significant 
resources are devoted to these activities. In contrast, DHS, as the lead for 
interagency homeland security efforts, is a new agency and has faced 
challenges since it was created due to frequent reorganization and not 
being fully staffed. 64 DHS and other civilian agencies and state and local 
governments have day-to-day missions and responsibilities that may take 
priority over exercises and often do not have the resources or experience 
to participate in or conduct exercises. For example, DOD exercises often 
are conducted 24 hours a day, 7 days a week and may last a week or more 
to enhance the realism of the exercise, while civilian agencies generally 
participate 8 hours per day, usually — according to NORTHCOM officials — 
during normal business hours, and do not exercise longer than a few days. 
Therefore, DOD exercises are generally longer in duration, more resource 
intensive, and involve more participants than other federal and state 
exercises. Furthermore, DOD views itself as the last line of defense and 
often exercises until resources are exhausted to fully assess capabilities 
and identify areas needing improvement. Civilian agencies and states may 
prefer not to exhaust resources during an exercise in order to avoid 
appearing unprepared for an incident and the associated political 
controversy. 



Defense Science Board, Unconventional Operational Concepts and the Homeland, viii, 
xiii, 38. 
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Another challenge that NORTHCOM faces is exercising with the various 
states and territories within its area of responsibility considering the legal 
and historical limits of the constitutional federal-state structure. The states 
have a wide range and type of civilian state agencies responsible for 
emergency management, some of which are headed by the Adjutant 
General of the state, who also heads the military department or National 
Guard, and others are completely separate entities (see table 4). 



Table 4: Emergency Management Structures of the States We Visited 



State 


Exercise(s) 


Emergency management structure 


Arizona 


Ardent Sentry 06 
Vigilant Shield 07 


The Adjutant General is the director of the Department of Emergency and Military 
Affairs, which is a state agency combining Arizona's Army and Air National Guard 
with Joint Military Programs and Emergency Management. 


California 


Vigilant Shield 09 
Determined Promise 4 


The California Emergency Management Agency is a Governor's cabinet-level 
agency that is led by a Secretary who reports to the Governor. 

The Adjutant General is the head of the California State Military Department 
under which the National Guard operates. 


Michigan 


Ardent Sentry 06 


The Civilian Emergency Management and Homeland Security Division is located 
in the State Police Department. 

The Adjutant General is the director of the Department of Military and Veteran 
Affairs under which the National Guard operates. 


Nevada 


Determined Promise 03 


The Division of Emergency Management is part of the Department of Public 
Safety (which also includes the Highway Patrol and State Fire Marshall). 

The Adjutant General is the head of the Nevada Office of the Military which is also 
known as the National Guard. 


Oregon 


Vigilant Shield 08 

National Level 
Exercise 1 -08 


The Oregon Office of Emergency Management operates administratively under 
the Adjutant General and the Oregon Military Department. The Adjutant General 
also directs, manages, and supervises the Oregon National Guard. The Office of 
Emergency Management's operations, however, are kept separate from those of 
the Oregon National Guard. 


Rhode Island 


Ardent Sentry 07 


The Adjutant General heads the Emergency Management Agency and the 
National Guard. These offices are co-located with the National Guard playing a 
supporting role to the Emergency Management Agency. 


Washington 


Ardent Sentry 08 

National Level 
Exercise 2-08 


The Adjutant General heads the Military Department which houses the 
Washington Emergency Management Division, National Guard, Air National 
Guard, and State Guard. 



Source: GAO analysis of NORTHCOM documents and information provided by states. 



Working with states has been the responsibility of the National Guard and 
is relatively new for a federal military command like NORTHCOM. 
NORTHCOM officials face challenges in dealing with the various civilian 
agencies, differing emergency management structures, capabilities, and 
needs of the states. For example, for Ardent Sentry 08 (linked with 
National Level Exercise 2-08), NORTHCOM planned a scenario involving a 
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chemical bomb attack in Seattle, Washington without consulting the state 
health department or civil support team — the agencies responsible for 
responding to a chemical or biological attack. State officials told us that 
NORTHCOM invited the health department to participate once state 
officials informed them that they should be involved, but that the scenario 
was already locked in without the input of this key participant. DOD 
officials told us that they rely on FEMA regional offices to provide 
information on state agencies. However, we believe that NORTHCOM 
officials should have determined if all relevant agencies were included in 
the exercise when directly interacting with state officials during the 
scenario development and other planning conferences, before the 
scenarios were locked in. Washington emergency management officials 
told us that this affected the realism of the exercise. 

NORTHCOM also faces challenges in balancing its training objectives with 
those of state agencies and organizations. State and local governments 
seek to exercise their first responder capabilities before having their 
resources overwhelmed and needing to seek federal assistance. On the 
other hand, NORTHCOM seeks to exercise its capability to provide 
support to civil authorities when local, state, and other federal resources 
are exhausted. This necessarily requires scenarios that exceed the states' 
capabilities and that stress DOD capabilities. Officials from four of the 
seven states we interviewed told us that NORTHCOM's exercise scenarios 
appeared unrealistic, overwhelmed their states too soon during the 
exercise, or did not allow states to fully exercise their own training 
objectives. 65 For example, the scenario for Ardent Sentry 06 included 
multiple improvised explosive devices detonating over a 4-day period in 
various sites, such as the City of Detroit, St. Clair and Wayne Counties, 
Michigan, and Windsor, Ontario, Canada with over 14,000 fatalities and a 
simultaneous pandemic flu outbreak in Michigan. 66 State emergency 
management officials told us that such a large number of casualties would 
overwhelm state resources almost immediately, and therefore precluded 



°Creating an exercise scenario event that is believable is critical to engaging participants in 
a level of play and experiential learning that will be long lasting. Defense Science Board, 
Unconventional Operational Concepts and the Homeland, 55. 

66 U.S. Northern Command, Northern American Aerospace Defense Command and United 
States Northern Command Ardent Sentry 06 Exercise Summary Report (Colorado 
Springs, Colo.: Sept. 27, 2006), 4-6. 
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fully exercising training objectives for state and local responders. 67 
Officials from these states told us that because they did not have the 
opportunity to exercise their own training objectives, they believed 
NORTHCOM was using them as a training tool. 68 A NORTHCOM official 
told us that NORTHCOM needs the states to participate in exercises and, 
therefore, will be flexible to accommodate other organizations' training 
objectives; however, NORTHCOM ultimately has its own objectives to 
exercise. Officials from five of seven states noted that, for example, they 
face budget and staffing limitations, and playing a major role in a 
NORTHCOM exercise often requires establishing a state emergency 
operations center with numerous staff and agencies involved. Given the 
expansive scenarios NORTHCOM uses to guide its exercises and the 
perception of half of the states we visited that this limits the benefits to 
them, we believe that the states may be less likely to expend scarce 
resources to participate in future NORTHCOM exercises. 

Inconsistencies with how NORTHCOM involves states in planning, 
conducting, and assessing exercises is occurring in part because 
NORTHCOM officials lack experience dealing with the various state 
agencies and emergency management structures. Inconsistencies are also 
occurring because NORTHCOM has not established an informed, 
consistent process for including states in its exercises. One aspect of this 
process is the way that NORTHCOM requests state participation in its 
exercises. Currently, NORTHCOM has various processes for requesting 
that other federal departments and agencies participate in its exercises, 
such as making the request through the Joint Staff. FEMA officials told us 
that requests for state participation in NORTHCOM exercises should be 
made through FEMA's regional offices. However, because NORTHCOM 
does not have an established process for requesting state participation, 
officials from the states we visited told us that NORTHCOM officials made 
requests informally and in a variety of ways, including through the 
National Guard Bureau, the state's National Guard, or FEMA's regional 



State emergency management officials told us that they attended planning meetings with 
NORTHCOM and raised their concern about the scenarios, but NORTHCOM officials made 
the decision to use the scenario anyway in order to meet their own training objectives. As a 
result, these officials told us that several other states dropped out of the exercise for this 
scenario during the planning stages. 

68 The Defense Science Board recently reported that national exercises typically focus on a 
top-down approach where the supporting organizations are training aids to the senior-level 
players, instead of a bottom-up approach focusing on an integrated and layered response. 
Defense Science Board, Unconventional Operational Concepts and the Homeland, xv. 
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offices. 69 In some cases, such as when the state emergency management 
agency and state National Guard have a close working relationship, this 
method has been effective for NORTHCOM. However, in other cases, this 
method has led to more limited exercises. For example, emergency 
management officials from one state told us that NORTHCOM does not 
have full state representation if it only exercises with the state National 
Guard. In that case, NORTHCOM therefore misses out on interaction with 
other key state emergency management officials and responders and 
affects the realism of the exercise. 

Another aspect of the lack of a consistent process for requesting state 
participation is potentially missing the opportunity to leverage the existing 
expertise of the National Guard Bureau and defense coordinating officers 
located in each of the 10 FEMA regional offices. As we previously 
reported, the National Guard Bureau and defense coordinating officers 
have knowledge and experience in dealing with states in their region and 
may be a valuable resource for NORTHCOM officials during the planning 
and conduct of exercises. 70 The three defense coordinating officers with 
whom we met told us that they participate in NORTHCOM exercises, but 
currently their role does not involve requesting state participation on 
behalf of NORTHCOM or providing state-specific information to 
NORTHCOM exercise officials. Without an informed and consistent 
process for including the states in planning, conducting, and assessing its 
exercises, NORTHCOM increases the risk that its exercises will not 
provide benefits for all participants, impacting the seamless exercise of all 
levels of government and potentially affecting NORTHCOM's ability to 
provide support to civil authorities. 



In some cases, NORTHCOM's coordination with the National Guard has been 
inconsistent. For example, National Guard officials from one state told us that it is unclear 
whether communication with NORTHCOM should occur directly or through the National 
Guard Bureau. 

'"Defense Coordinating Officers are senior-level military officers who provide liaison 
support and serve as a point of contact for federal, state, and local agencies requiring DOD 
support during an incident. GAO-08-252. 
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NORTHCOM has a 
Systematic Lessons 
Learned and 
Corrective Action 
Program, but Gaps 
Remain 



NORTHCOM Has a 
Systematic Lessons 
Learned and Corrective 
Action Process 



DOD and NORTHCOM guidance requires that NORTHCOM identify 
observations during the course of normal operations, exercises, and real- 
world events; capture the detail required to fully understand the problem; 
and share valid lessons learned and issues as widely as possible. 71 
NORTHCOM has been identifying observations, lessons learned, and 
needed corrective actions from its exercises and operations since the 
command was created in 2002. NORTHCOM collects and tracks 
observations through the Joint Lessons Learned Information System 
(JLLIS) — the automated official DOD system for managing and tracking 
exercise observations and recording lessons learned. 72 As of April 2009, 
DOD exercise participants input 94 observations into JLLIS during 
NORTHCOM's most recent large-scale exercise, Vigilant Shield 09. Table 5 
shows the observations entered into JLLIS or its predecessor for 
NORTHCOM's major exercises since 2006. 73 



Chairman of the Joint Chiefs of Staff, Joint Lessons Learned Program, Instruction 
3150.25D, A-l and A-6; U.S. Northern Command, Commander's Training Guidance FY 09- 
10, A-4, A-8, A-20, A-2-8: U.S. Northern Command, Lessons Learned Program and 
Corrective Action Board Process, 2. 

72 The Joint Staff develops joint lessons learned policy and serves as the program manager 
for JLLIS. 

73 JLLIS became the system of record in October 2008. Prior to that NORTHCOM's lessons 
learned system was the Advanced Lesson Management System. 
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Table 5: Number of Observations Input into Lessons Learned Management System for Large-Scale Exercises Since 2006 


NORTHCOM exercise 


Unclassified 
observations" 


Classified 
observations 


Total 
observations 
per exercise 


Vigilant Shield 09 


75 


19 


94 


Ardent Sentry 08 

(National Level Exercise 2-08) 


46 





46 


Vigilant Shield 08 

(National Level Exercise 1-08) 


34 





34 


Ardent Sentry 07 


113 


4 


117 


Vigilant Shield 07 


95 


12 


107 


Ardent Sentry 06 


54 


2 


56 


Vigilant Shield 06 


12 


1 


13 



Source: Joint Lessons Learned Information System 



Note: This includes active records on the Non-Secure Internet Protocol Router Network as of April 9, 
2009; and on the Secure Internet Protocol Router Network (classified and unclassified) as of May 4, 
2009 

includes unclassified active records from the Secure Internet Protocol Router Network. 

The philosophy and approach of NORTHCOM's Lessons Learned Program 
have been largely the same since NORTHCOM published its first 
instruction for the program in 2003, 74 although the requirement to re- 
observe corrective actions in a subsequent exercise or operation before 
closing them was not established until 2005. 75 We found that NORTHCOM 
generally has a systematic lessons learned and corrective action program, 
based on clear procedures and a regular process. Observations are 
assigned to an office of primary responsibility within NORTHCOM and 
categorized as either a lesson learned — a positive finding — or an issue 
which requires corrective action. NORTHCOM's intent is to manage and 
resolve issues requiring corrective action at the lowest organizational level 
possible. This responsibility is generally within NORTHCOM's various 
directorates, component commands, or a Joint Task Force. 76 Issues may 
be closed at the directorate level without external approval or oversight. 



U.S. Northern Command, Lessons Learned Policy and Guidance, Operating Instruction 
36-104, (Colorado Springs, Colo.: Oct. 1, 2003). 

75 Chairman of the Joint Chiefs of Staff, Instruction 3150. 25B, Joint Lessons Learned 
Program, A-2 -A-3 (Washington, D.C.: February 15, 2005). 

76 Those component commands, such as Army Forces North or Air Force North, may also 
have a lessons learned program of their own. 
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Broader scope or more sensitive issues requiring the involvement of more 
than one directorate or subcommand go into the formal corrective action 
board process for review, tracking, and approval as necessary. 77 This 
formal process includes two boards — the Corrective Action Board and the 
Executive Corrective Action Board — to review and resolve issues. 78 Figure 
7 illustrates the flow of NORTHCOM's lessons learned and corrective 
action process. 



The Corrective Action Board process is intended to provide a means for addressing 
commandwide issues and ensuring these issues are tracked, resolved, and verified. 

,8 The Corrective Action Board Working Group is chaired by the head of the Joint Training 
and Exercises Directorate, and includes representatives from each NORTHCOM 
directorate and subcommand and reviews issues requiring extensive coordination across 
directorates and subordinate commands or action by an organization(s) external to 
NORTHCOM. The Executive Corrective Action Board is chaired by NORTHCOM's Chief of 
Staff and comprised of the heads of the directorates and subcommands and reviews 
sensitive issues requiring decisions by senior leaders or actions of organizations outside of 
NORTHCOM, such as the Joint Staff. For example, lessons learned that are shared with 
DHS are reviewed by the Executive Corrective Action Board before they are forwarded to 
the Joint Staff. 
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Figure 7: NORTHCOM Lessons Learned and Corrective Action Process 



Observation 
recorded 



Responsible 
office 
assigned 




Source: GAO analysis of NORTHCOM guidance. 
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Gaps Remain with 
Collecting Observations 
From and Sharing Lessons 
Learned with Interagency 
Partners and States 



JLLIS was intended to make lessons learned more widely available; 
however, we found that non-DOD exercise participants have limited 
access to this system, which presents challenges for NORTHCOM in 
collecting observations from and sharing lessons learned with interagency 
partners and states. 79 Joint Staff and NORTHCOM officials told us that 
only a small number of interagency staff and no state representatives 
currently have access to JLLIS and therefore are not able to directly input 
exercise observations or review NORTHCOM's lessons learned through 
this system. 80 DOD officials told us that access to JLLIS is restricted in 
order to safeguard sensitive information because of concerns that 
adversaries may be able to exploit any weaknesses contained in those 
records. On the other hand, NORTHCOM officials told us that most of the 
observations collected from exercises are unclassified, and we found this 
to be true for more than 90 percent of the observations from 
NORTHCOM's six most recent exercises. The vast majority of the 
observations we reviewed are listed as unclassified and few records 
indicate they have a "For Official Use Only" restriction. For users to access 
the unclassified portion of the JLLIS they must have a DOD common 
access card, a card reader, and appropriate computer software prior to 
requesting access. Special security arrangements must be made for 
granting JLLIS access to civilian federal officials who do not possess 
common access cards but have other federally issued personal 
identification verification cards. These officials cannot access JLLIS 
because they also lack the appropriate software to operate the personal 
identification verification cards within the system. In an attempt to 
address this gap and gather more input from non-DOD exercise 
participants, in April 2008 NORTHCOM officials placed a Word document 
template on its internet portal to provide a venue for non-DOD exercise 
participants to record exercise observations. Any observations provided in 
this format would be input by NORTHCOM Joint Training and Exercise 



NORTHCOM has a process for sharing lessons learned from National Level Exercises that 
are applicable to the interagency to DHS through the Joint Staff. Thus far, NORTHCOM has 
provided three3 issues requiring corrective action from National Level Exercise 2-08 
conducted in May 2008. 

80 Joint Staff officials told us that about lODepartment of State officials have access to 
JLLIS, and DHS and Department of Energy officials have requested access for some of their 
staff. Other departments and agencies, such as the Federal Bureau of Investigation and 
Department of Health and Human Services, have received briefings about JLLIS, but have 
not requested access. The Joint Staff official also told us that his office does not have 
enough staff to support a large number of non-ommon access card users requesting JLLIS 
access, and granting access would be a lengthy process due to the software and security 
requirements that must be addressed. 
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Directorate officials into JLLIS. However, NORTHCOM's portal requires 
the same DOD-issued card to gain entry. 81 In addition, NORTHCOM's 
lessons learned manager told us no one has submitted observations using 
the template since it was put on the portal. This may be because the 
command has not actively publicized how to access the template and 
underscored the value to the command of obtaining observations from 
interagency partners and states. In response to our inquiries in May 2009, 
NORTHCOM's lessons learned manager told us that the command is in the 
process of adding a link to DHS's Homeland Security Information Network 
so that interagency partners and states will be able to submit lessons 
learned which can subsequently be transferred to JLLIS by NORTHCOM 
officials. 82 

In addition to collecting observations using JLLIS, NORTHCOM can obtain 
lessons learned from interagency partners and states during postexercise 
meetings. NORTHCOM conducts a review called a Hotwash within hours 
of completing the exercise so that exercise participants can discuss 
observations that significantly impacted their mission and recommend 
emergent themes for discussion during a subsequent review known as the 
facilitated after-action review. This review, generally held 7 days after the 
exercise is completed, provides an opportunity to present major issues to 
senior leaders and obtain the Commander's guidance for resolution. 
However, the extent to which interagency and state officials are attending 
and participating in NORTHCOM's postexercise meetings is unclear. 
Based on NORTHCOM's documentation, only two states (out of the last 
six major exercises) participated in a Facilitated After Action Review — 
California in Vigilant Shield 09 and Alaska in Ardent Sentry 07. Officials 
from three states in addition to California told us that they participated in 
the after-action meeting for the exercises they participated in, but they 
may have participated in the regional or national-level meeting rather than 
NORTHCOM's. 83 Officials from two of the seven states we met with told us 
that they did not attend NORTHCOM's postexercise reviews for the 



NORTHCOM's lessons learned manager told us that those without cards that have a valid 
need for access to the portal must work with NORTHCOM's Help Desk staff to register for 
an account. 

82 DHS's Web-based Homeland Security Information Network facilitates information sharing 
and collaboration between federal, state, local, tribal, private sector, and international 
partners involved in homeland security. 

83 Depending on the exercise, for example if it is linked to a national level or regional 
exercise, multiple Facilitated After Action Reviews may take place. 
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exercises in which they participated at least partly due to staffing and 
budget limitations. 

NORTHCOM has also attempted to share lessons learned with other 
federal agencies and states by using FEMA's lessons learned sharing 
system. For example, NORTHCOM has posted six reports onto FEMA's 
lessons learned system, including four recent exercise reports and two 
reports from operations in 2008. 84 However, with one exception, the 
documents that NORTHCOM has made available on this system (1) 
include only lists of observations and, in some cases, record-tracking 
numbers from JLLIS and previous lessons learned systems, and (2) lack 
detailed information on individual lessons learned and corrective actions. 8i 
Joint Staff and NORTHCOM officials told us that they do not post detailed 
information on the unclassified Lessons Learned Information Sharing 
system Web site, because it is not adequately protected from the potential 
for unauthorized access to records. As a result, the security of the 
information cannot be assured. According to these officials, if an 
adversary nation or terrorist group gained access to this information, it 
may be possible for them to identify weaknesses in NORTHCOM's 
operations that can be exploited. In a recent exercise summary report 
NORTHCOM stated that it will post lessons learned, best practices, and 
reports that may benefit their non-DOD mission partners in FEMA's 
Lessons Learned Information Sharing system, which the report describes 
as a secure, restricted-access information system. 86 Because security 
concerns are preventing NORTHCOM from openly sharing all its 
unclassified lessons learned with its interagency partners and the states, 
the information NORTHCOM does provide may be of limited value for 
helping its partners improve the nation's disaster responsiveness. Because 
NORTHCOM is not fully involving other federal agencies and states in its 
lessons learned process, it is missing opportunities to learn lessons from 
an exercise. For example, officials from two states did not provide 
NORTHCOM with lessons learned from exercises because they did not 



NORTHCOM reports posted on FEMA's Lessons Learned Information Sharing system 
include exercise summary reports for National Level Exercise 1-08 (Vigilant Shield 08), and 
Ardent Sentry 2007; two copies of the Quick Look Report for National Level Exercise 2-08; 
NORTHCOM Support for 2008 Hurricane Season Operations Summary Report; and 
NORTHCOM Support for 2008 Political Conventions Operations Summary Report. 

85 We found one document where details were provided on 12 observations. 

86 U.S. Northern Command, National Level Exercise 2-08 Exercise Summary Report 
(Colorado Springs, Colo.: Aug. 15, 2008), 16. 
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attend the command's post exercise reviews. As a result, NORTHCOM 
risks the reoccurrence of potential problems that were not identified in its 
process. 



NORTHCOM Faces 
Challenges Managing 
Corrective Actions 



DOD and NORTHCOM guidance requires that issues requiring corrective 
actions be tracked and remain open until the solutions are completed and 
verified as effective — through training, operations, or exercises. 87 We 
found that NORTHCOM directorates and subcommands are closing some 
issues prematurely, without confirming that corrective actions were made 
or verifying in a subsequent exercise or operation that the corrective 
action is effective. We reviewed unclassified records in JLLIS from 
NORTHCOM's previous six large-scale exercises and found at least 77 of 
the 375 records or about 20 percent required corrective actions but were 
either closed prior to completing the corrective action or closed without 
verifying the effectiveness of the corrective action. 88 For example, an 
observation was made during Ardent Sentry 07 that NORTHCOM did not 
have a process for addressing a foreign nation's offer of military-to-military 
assistance in a major disaster. The issue was validated and the corrective 
action developed, but the issue was closed by the originating organization 
before the corrective action could be verified or reobserved in a 
subsequent exercise. 89 The record was closed even though the Executive 
Corrective Action Board directed that it remain open until an exercise of 
suitable scope to require significant military support was developed. 90 
Another example of a record being closed without verification or re- 
observation is an observation made during Ardent Sentry 07 raising 
concerns that NORTHCOM personnel could arrive to assess a disaster site 



Chairman of the Joint Chiefs of Staff, Instruction 3150. 25D, Joint Lessons Learned 
Program (Washington, D.C.: Oct. 10, 2008), A-8; U.S. Northern Command, Lessons Learned 
Program and Corrective Action Board Process, Instruction 16-166 (Colorado Springs, CO: 
Mar. 16, 2009), 14; Prior Chairman of the Joint Chiefs of Staff guidance also required that 
corrective actions be tracked and remain open until the solutions are completed and 
verified. 

88 We reviewed active records from NORTHCOM's large-scale exercises that were 
conducted between May 2006 and November 2008 for Ardent Sentry 2006, 2007, and 2008 
and Vigilant Shield 2007, 2008, and 2009. 

89 The corrective action included a set of checklists and processes added to the command's 
battle staff operating procedures. 

90 The scenarios developed for NORTHCOM's next two applicable major exercises were not 
of sufficient scope to induce another nation, other than Canada, to offer military 
assistance. 
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without alerting state officials they would be coming. As a result, 
NORTHCOM developed a new Command Assessment Element Concept of 
Execution 91 in July 2007 to promote better command and control and 
situational awareness; however, the issue was closed before the procedure 
could be observed in a subsequent exercise or operation to verify effective 
resolution. 

These issues are likely being closed without verification or re-observation, 
because NORTHCOM Training and Exercise Directorate officials do not 
have oversight over the disposition of open issues that are resolved within 
directorates or are unable to give long-standing issues the sustained 
management attention needed to ensure resolution. NORTHCOM's lessons 
learned manager told us that the command does not have the staff 
necessary to oversee the actions on records handled within the other 
directorates. In addition, while the checkbox format in JLLIS makes it easy 
to see whether an issue is open, awaiting verification, or closed, entries 
made in JLLIS regarding corrective actions required, implementation date, 
and plan for verification are primarily in a narrative format, which may 
make the review and oversight process more time consuming. Without 
sufficient oversight, NORTHCOM cannot ensure that corrective actions 
are verified and reobserved in a subsequent exercise or operation before 
the issue is closed, so that the command knows the solution is effective. 
We recognize that such oversight should be addressed without 
significantly stressing NORTHCOM's staff. However, if NORTHCOM does 
not ensure that corrective actions are fully resolved, it increases the risk 
that these issues may occur again, possibly during crucial, real-world 
situations. This lack of oversight, coupled with the lack of a well-thought 
out and consistent process for including the states in assessing exercises 
as discussed earlier in this report, further limits the knowledge gained and 
value of the exercise for all participants. 



The Command Assessment Element is a team supporting NORTHCOM's commander by 
providing situational awareness from a disaster or incident site. 
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NORTHCOM Has 
Taken Steps to 
Integrate Its 
Exercises with the 
National Exercise 
Program, but 
Guidance Is Not 
Consistently Applied 

Since the NEP Charter was approved in January 2007, NORTHCOM has 
participated in the major national exercise held under the NEP and taken 
steps to integrate its exercises into the national program. 92 NEP guidance 
requires that heads of departments and agencies actively participate in tier 

1 exercises and recommends participation in tier II exercises either 
through the National Exercise Simulation Center or as determined by 
agency leadership. 93 Departments or agencies can participate in the NEP 
by combining an existing exercise with a NEP exercise, taking part in a 
tier II exercise sponsored by a different department or agency, or 
requesting to lead a tier II exercise to obtain greater interagency 
participation and support. DOD guidance requires that components 
participate in or lead planning efforts of NEP exercises as appropriate 
given the scenario or as tasked by the Assistant Secretary of Defense, 
Homeland Defense and America's Security Affairs or the Chairman of the 
Joint Chiefs of Staff. 94 NORTHCOM's training guidance specifies the NEP 
exercises in which the command plans to participate during the following 

2 fiscal years. 95 NORTHCOM combined two of its large-scale exercises — 
Vigilant Shield 08 and Ardent Sentry 08 — with major national exercises 



For the purpose of this report, we are defining major national exercises as operations- 
based NEP exercises involving the deployment of personnel or forces to conduct 
operations from the federal, state, and local levels. 

93 DHS, National Exercise Program Implementation Plan. 

94 DOD, DOD Participation in the National Exercise Program, Instruction 3020.47, 
(Washington, D.C.: Jan. 29, 2009), 10. 

95 The command's Joint Training Plan, which includes the commander's training guidance, is 
revised and updated annually. U.S. Northern Command, Commander's Training Guidance 
for FY 09-10. 



NORTHCOM Has 
Participated in National 
Exercises and Taken Steps 
to Integrate with the NEP 
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and has taken part in two additional exercises sponsored by other 
departments (see table 6). For example, National Level Exercise 1-08, a 
tier I exercise, and NORTHCOM's Vigilant Shield 08 were conducted 
October 15 -20, 2007, in parallel with Top Officials 4 96 and several other 
exercises. 97 These exercises were linked together by the use of common 
scenarios and objectives intended to test existing plans, policies, and 
procedures to identify planning and resource gaps and develop corrective 
actions to improve preparedness against a weapons of mass destruction 
attack. 



Table 6: NORTHCOM Participation in National Level and Tier II Exercises Since 2007 


Exercise and tier 


Date 


Main scenario 


Primary sponsor/lead 


Description of 

NORTHCOM 

participation 


National Level 
Exercise 1-08 (I) 


October 2007 


Radiological dispersal device 
attack 


FEMA 


Linked Vigilant Shield 08 
to exercise 


National Level 
Exercise 2-08 (II) 


May 2008 


Hurricane, terrorist chemical 
attack, accidental chemical 
release 


FEMA 


Linked Ardent Sentry 08 
to exercise 


Diablo Bravo (II) 


July 2008 


Nuclear weapons incident 


Department of Energy/ 
National Nuclear Security 
Administration 


Provided initial DOD 
response support for 
Department of Energy 


Empire 2009 (II) 


June 2009 


Radiological dispersal device 
Attack 


National Nuclear Security 
Administration 


Provided Defense 
Coordinating Officer 
support 



Source: GAO analysis of DHS and NORTHCOM documentation. 

Note: We limited the scope of our finding to major national exercises sponsored at the federal level 
and planned after National Level Exercise 1 -08, the first major exercise conducted under the NEP. 



NORTHCOM officials told us that they generally would like to participate 
in NEP exercises to achieve the benefits of exercising with interagency 
partners, but in some cases it is not beneficial to do so. For example, the 
officials told us NORTHCOM decided not to combine Ardent Sentry 09 



The Top Officials exercise, better known as TOPOFF, is the nation's terrorism 
preparedness exercise involving officials at every level of government and representatives 
from the international community and private sector. Sponsored by DHS, TOPOFF 4 
shared the same radiological dispersal device scenario in Vigilant Shield 08 and included 
more than 100 organizations. 

97 National Level Exercise 1-08 included Global Lightning 08 at U.S. Strategic Command, 
Positive Response 08-1 for the Chairman, Joint Chiefs of Staff, Blue Flame in the United 
Kingdom, Pacific Challenge in Australia, a U.S. Southern Command Response Cell due to 
Able Warrior 08-1 date change, and an unnamed exercise at U.S. Pacific Command. 
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with National Level Exercise 09 — a tier 1 exercise scheduled for July 
2009 — because the objectives and scenarios for the exercises did not meet 
their training needs. Although NORTHCOM officials will conduct Ardent 
Sentry 09 separately, they are using the National Exercise Simulation 
Center — FEMA's newly established training and exercise facility — to 
provide a test run for the center's use in National Level Exercise 09. 98 

DOD and NORTHCOM have taken steps to integrate exercises with the 
National Exercise Program, including posting the command's exercises on 
DHS's National Exercise Schedule, successfully applying to lead a tier II 
exercise, and publishing guidance on integration with the NEP. The NEP 
Implementation Plan recommends that federal departments and agencies 
post exercises on the NEP's National Exercise Schedule so that exercises 
and planning meetings can be synchronized across the federal 
government." NORTHCOM has posted its annual Ardent Sentry and 
Vigilant Shield exercises for the first 4 of 5 fiscal years on the national 
schedule, while FEMA's National Exercise Division has posted exercises 
for the first 3 fiscal years. As of June 2009, neither the Joint Staff nor any 
other combatant commands have posted exercises on the national 
schedule. In addition, NORTHCOM recently requested and was granted 
approval to lead Vigilant Shield 10 as a tier II exercise scheduled for 
November 2009. Vigilant Shield 10 should have greater interagency 
participation than it would have received as a tier 3 exercise, since federal 
departments and agencies will be required, at a minimum, to participate in 
the National Exercise Simulation Center. As of May 2009, the participants 
of Vigilant Shield 10 include the DHS and the Departments of Justice, 
Energy, Transportation, Health and Human Services, and Veteran Affairs; 
the Joint Chiefs of Staff and U.S. Joint Forces Command; and other 
government and nongovernment organizations. 100 This exercise will be the 
first time that NORTHCOM will share planning responsibilities with 
FEMA's National Exercise Division. This exercise will also be linked to a 



FEMA's National Exercise Simulation Center opened on January 12, 2009, to provide a 
national, shared facility to train and exercise and improve coordination for all-hazards 
preparedness. The goal was to pool resources, maximize efficiency, and provide sustained 
exercise and training support to all participants. 

"FEMA officials told us that only tier 1 and 2 exercises are required to be included in the 
national schedule, but encouraged the inclusion of Tier 3 exercises. 

100 The list of participating departments or agencies may change as the planning for Vigilant 
Shield 10 progresses. 
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Canadian government an exercise to demonstrate its readiness for the 
2010 Olympics in Vancouver. 



NORTHCOM Has Not NEP guidance includes policies and tools for the design, planning, 

Consistently Applied NEP conduct, and evaluation of exercises — known as the Homeland Security 
Guidance Exercise and Evaluation Program, which creates a common exercise 

policy and consistent terminology for exercise planners and serves as the 
foundation of NEP exercises. 101 FEMA requires that entities, such as state 
and local governments, receiving homeland security grant funding for their 
exercises adhere to specific Homeland Security Exercise Evaluation 
Program guidance for exercise program management, design, conduct, 
evaluation, and improvement planning. 102 We reviewed key program 
documents, such as the Implementation Plan, and found that this guidance 
is unclear about the extent to which federal agencies should use the 
Homeland Security Exercise Evaluation Program. For example, the 
Implementation Plan states that the NEP does not displace a preexisting 
exercise program, and none of the NEP guidance requires that federal 
agencies use the Homeland Security Exercise Evaluation Program. 
However, the Implementation Plan states that the Homeland Security 
Exercise Evaluation Program will serve as the doctrinal foundation for 
NEP exercises. 

FEMA officials told us that federal agencies should use this program when 
participating in tier I and tier II exercises so that the various exercise 
participants have consistency when planning, conducting, and assessing 
exercises. We found that NORTHCOM generally has used DOD's Joint 
Training System guidance for planning NEP exercises, defining 
capabilities, and reporting exercise results. 103 NORTHCOM officials told us 
that the Joint Training System is consistent with the NEP and served, in 
part, as the basis for the Homeland Security Exercise and Evaluation 



This plan was revised in June 2008, but has not been formally approved by the current 
administration. However, FEMA officials told us that they intend to execute the NEP using 
the latest revision of the plan. DHS, DRAFT National Exercise Program Implementation 
Plan (Washington, D.C.: June 20, 2008). 

102 GAO-09-369, p. 35. 

103 NORTHCOM officials told us that they have used Homeland Security Exercise Evaluation 
Program guidance in one case when they drafted the exercise plan for NLE 2-08, because 
their state and local exercise partners were required to use this guidance. 
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Program. 104 We found that these sets of guidance have similar processes 
but use different methods for defining the tasks and capabilities that are 
performed and validated in an exercise. The primary differences between 
these sets of guidance are that (1) DOD's task list, which serves as the 
basis for its exercises, includes tasks that are specific to military missions, 
such as troop movements and sealifts; (2) DHS guidance provides more 
detailed criteria for the postexercise documentation, such as content and 
format; and (3) DHS's planning cycle is generally shorter — 9 to 15 months 
versus 12 to 18 months for DOD. (See table 7.) See app. II for a more 
detailed comparison. 



Table 7: Comparison of Areas of Application between DOD's Joint Training System and DHS's Homeland Security Exercise 
Evaluation Program 



Areas of application 


Joint Training System 


Homeland Security Exercise Evaluation Program 


Methods for defining tasks 
and capabilities 


Derived from Joint Mission Essential Task 
Lists. 

Users may also develop tasks using the 
Universal Joint Task List. 


Derived from Target Capabilities List or 
Universal Task List. 

Users may also develop tasks using the 
Universal Task List. 


Reporting requirements 


Issue a Quick Look Report and an 
Executive Summary Report. 

No detailed guidelines on content for either 
report. 

No detailed guidelines on content found in 
the required Quick Look or Exercise 
Summary Report. 


Issue a Quick Look Report and an After Action 
Report/Improvement Plan. 

Provides detailed guidelines on content for both 
reports, including templates. 

Provides detailed guidelines on content for both 
the required Quick Look Report and the required 
After Action Report/Improvement Plan. 


Planning cycle 


Uses the Joint Event Life Cycle, has five 
stages: design, planning, preparation, 
execution, and evaluation. 

Intended to be flexible and can be modified 
to any type of event. Can span a period of 
12 to 18 months for exercises. 


Uses the Homeland Security Exercise Evaluation 
System's Exercise Cycle, employs five phases: 
foundation, design and development, conduct, 
evaluation, and improvement planning. 

Intended to be adaptable to any type of exercise. 
Can span a period of 9 to 15 months. 


Lessons learned/corrective 
actions systems 


Joint Lessons Learned Information System 
manages and tracks Lessons Learned and 
corrective actions. 


FEMA's Lessons Learned Information Sharing 
System manages and disseminates lessons 
learned. 



DHS's Corrective Action Program system 
manages corrective actions. 

Source: GAO analysis of DOD and DHS documentation. 



104 NORTHCOM's training guidance for fiscal years 2008-2009 stated that the command will 
adhere to the guiding principles of the Homeland Security and Exercise Evaluation 
Program, but this reference has been removed from guidance for fiscal years 2009-2010. 
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According to both the Homeland Security Exercise and Evaluation 
Program and Joint Training System guidance, it is important to link tasks 
and capabilities with exercise objectives to ensure that participants 
exercise or train as they would perform in a real-world event. The 
Homeland Security Exercise and Evaluation Program recommends using 
DHS's Target Capabilities List or the Universal Task List to formulate the 
tasks and capabilities that underlie the objectives for an exercise. 105 These 
lists describe the capabilities government entities need and tasks they are 
expected to perform to prevent, protect against, respond to, and recover 
from incidents of national significance. In contrast, NORTHCOM derives 
its tasks and capabilities from the Universal Joint Task List to formulate 
Joint Mission Essential Tasks. According to NORTHCOM guidance, the 
command is required to include in its exercises the Joint Mission Essential 
Tasks associated with its Joint Training Plan, which is updated annually. 106 
These tasks are identified by joint force commanders as most essential to 
their assigned or anticipated missions with priority given to their wartime 
missions and describe their priority wartime requirements. We found that 
DOD's operating instruction for participation in the NEP does not provide 
guidance on how DOD components should incorporate tasks and 
capabilities derived from sources recommended by the Homeland Security 
Exercise and Evaluation Program when participating in NEP exercises. 107 
The primary differences between DHS' and DOD's lists are that DOD's task 
lists generally incorporate more descriptive metrics and criteria to assess 
performance and include tasks that are specific to military missions, such 
as troop movements and sealifts. In some cases, state National Guards 
officials have had to translate DOD task lists into DHS tasks lists when 
working with their civilian partners and vice versa. 

We also found that neither DOD's nor NORTHCOM's guidance for 
developing postexercise reports includes the same degree of specificity 
recommended in the Homeland Security Exercise Evaluation Program. 
For example, both sets of exercise guidance require postexercise reports; 
however, the Homeland Security Exercise Evaluation Program provides 
templates and guidance for these documents, including requiring an 
improvement plan to clearly outline the corrective actions needed, which 



DHS, Homeland Security Exercise and Evaluation Program, Volume 1: HSEEP 
Overview and Exercise Program Management (Washington, D.C.: February 2007). 

106 U.S. Northern Command, Operations: Exercise Program, pp. 4-5. 

107 DOD, DOD Participation in the National Exercise Program, Instruction 3020.47, 7-8. 
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are not included in DOD's or NORTHCOM's guidance. In addition, 
NORTHCOM's exercise summary reports for National Level Exercise 1-08 
and 2-08 did not contain all information recommended by the Homeland 
Security Exercise Evaluation Program. For example, NORTHCOM did not 
include the recommended analyses regarding the capabilities and tasks 
tied to the exercises' objectives. 

As stated above, we reviewed NEP guidance such as the Implementation 
Plan and found it does not clearly state the extent to which federal 
agencies are required to follow the Homeland Security Exercise and 
Evaluation Program. As a result of this unclear guidance, we found that 
agency officials have varying interpretations of the requirements. For 
example, a DOD and a Joint Staff official told us that NEP guidance does 
not require agencies to use the Homeland Security Exercise Evaluation 
Program even for NEP exercises. Therefore, NORTHCOM uses the Joint 
Training System rather than the Homeland Security Exercise and 
Evaluation Program as the basis for planning, conducting, and assessing 
exercises. However, officials from FEMA's National Exercise Division told 
us that all participating agencies should use the Homeland Security 
Exercise and Evaluation Program guidance for tier I and tier II NEP 
exercises. 108 FEMA officials stated that federal departments and agencies 
should be held accountable for meeting key requirements, but that FEMA's 
authority is limited to guiding, supporting, and coordinating with, but not 
directing other federal departments and agencies to comply with guidance. 
As we have previously reported, we believe that FEMA's expanded 
leadership role under the Post-Katrina Act provides FEMA opportunities 
to instill a shared sense of responsibility and accountability on the part of 
all agencies. 109 

Neither DOD nor NORTHCOM guidance specifically addresses the extent 
to which DHS's Homeland Security Exercise and Evaluation Program 
planning and documentation requirements should be followed. 110 We 
recognize that NORTHCOM and DOD must meet their own mission and 



According to FEMA officials, since tier I and II exercises are interagency exercises, the 
Homeland Security Exercise and Evaluation Program should be used because it was 
designed for the interagency community. 

109 GAO-09-369. 

110 NORTHCOM guidance states the Vigilant Guard Program should use Homeland Security 
Exercise and Evaluation Program formats to encourage civilian agency participation. U.S. 
Northern Command, Commander's Training Guidance: Fiscal Year 2009-2010, A-18. 
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exercise requirements and the Joint Training System may be best suited 
for NORTHCOM's exercises; however, all of the states we visited use 
Homeland Security Exercise and Evaluation Program guidance. We found 
that having differing sets of guidance, such as DOD's and DHS' capabilities 
task lists and postexercise documentation requirements, makes exercises 
more difficult and potentially limits the benefits for participating states. 
For example, officials from three states we visited told us that using 
NORTHCOM's exercise planning and reporting requirements rather than 
Homeland Security Exercise Evaluation Program guidance has made the 
processes more difficult. Further, the Defense Science Board found that 
inconsistent approaches to the development and content of postexercise 
documentation may affect the ability of organizations to fully learn lessons 
identified in exercises. 111 We also reported that when other federal entities 
carry out processes that do not specifically follow the Homeland Security 
Exercise and Evaluation Program, FEMA managers do not have the 
necessary data to measure progress, identify gaps in preparedness, and 
track lessons learned — key objectives of the NEP. 112 We believe that 
achieving national preparedness requires a whole-of -government approach 
and is a shared responsibility among federal, state, local, and tribal 
governments and organizations and an integration of their various 
standards, policies, and procedures into the national system. 11:3 



There is an increasing realization within the federal government that an 
effective, seamless national response to an incident requires a strong 
partnership among federal, state, and local governments and 
organizations, including integrated planning, training, and the exercise of 
those plans. For DOD, the effective execution of civil support, especially 
amid simultaneous, multijurisdictional disasters, requires ever-closer 
working relationships with other departments and agencies and at all 
levels of government. NORTHCOM's use of DOD's Joint Training System 
has provided a robust process for planning and conducting exercises to 
improve preparedness to achieve its homeland defense and civil support 
missions, and its efforts to involve its interagency partners and the states 
in exercises have helped to reduce uncertainty about the process for 
responding to an incident. However, without a consistent record of what 



Defense Science Board, Unconventional Operational Concepts and the Homeland. 
: GAO-09-369, p. 42. 
'GAO-09-369, p. 68. 
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has occurred during an exercise that is accessible by all exercise 
participants, including those from other federal agencies and states, 
NORTHCOM cannot ensure that it has met internal standards, trained to 
key focus areas, or compared the goals and results of exercises over time. 
Further, a key element to developing effective working relationships with 
all states is a consistent process for including states in planning and 
executing NORTHCOM's exercises that incorporates state-specific 
knowledge and information. By coordinating consistently with 
organizations, like FEMA and NGB, that have knowledge and experience 
dealing with states, NORTHCOM can improve the value and effectiveness 
of exercises for all of the participants involved. 

Exercises provide an opportunity to enhance preparedness by collecting, 
developing, implementing, and disseminating lessons learned and verifying 
corrective action taken to resolve previously identified issues. 
NORTHCOM's clear procedure for capturing observations in JLLIS and 
identifying issues needing corrective action has helped to improve its 
capabilities to complete its missions. However, by not providing federal 
agencies and states greater access to its lessons learned process, 
NORTHCOM will lose opportunities to learn valuable lessons from an 
exercise, particularly observations from the states that could enhance 
coordination and build more effective interagency relationships. Further, 
the risk that issues may reoccur will be increased, particularly when 
interagency partners are not aware of key issues or concerns that might 
impede the government's overall responsiveness to a natural or man-made 
disaster. In addition, when corrective actions remain open until fully 
implemented and verified in a subsequent exercise, NORTHCOM will have 
greater assurance that issues raised during exercises are being adequately 
addressed and the corrections are in fact solving the problems identified. 

NEP policies and tools for the design, planning, conduct, and evaluation of 
exercises are intended to create a common exercise policy and consistent 
terminology for exercise planners across all levels of government to 
improve the federal government's ability to evaluate national 
preparedness. The steps DOD and NORTHCOM have taken to integrate 
exercises with the NEP have helped DHS to prioritize and coordinate 
federal exercise activities and enhance the federal government's ability to 
respond to an incident. We recognize that NORTHCOM and DOD must 
meet their own mission and exercise requirements and the Joint Training 
System may be best suited to meet the high standards required for 
NORTHCOM's exercises. However, achieving national preparedness 
requires shared responsibility among federal, state, and local governments 
and organizations and an integration of their various standards, policies, 
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and procedures into the national system. We also recognize that the NEP 
continues to evolve and become more useful to federal and state partners. 
IHowever, in the absence of clear guidance from DHS on the extent to 
which agencies should use Homeland Security Exercise Evaluation 
Program planning and documentation guidance, DOD should ensure that 
its components clearly understand when the use of this guidance is 
appropriate so that both DOD and its exercise partners, such as other 
federal agencies and states, derive the most benefits from exercises. This, 
in turn, contributes to the ultimate success of a whole-of-government 
approach to national preparedness. 



Recommendations for 
Executive Action 



To improve NORTHCOM's consistency with exercise documentation, we 
recommend that the Secretary of Defense direct NORTHCOM's 
Commander to develop guidance with specific criteria for postexercise 
documentation, particularly the Exercise Summary Report as the official 
exercise record, including the content and format to be included in such 
reports that will allow the results and lessons learned of exercises to be 
easily reviewed and compared. 



To improve NORTHCOM's involvement of interagency partners and states 
in its exercises, we recommend that 

• the Secretary of Defense, in consultation with the Chairman of the 
Joint Chiefs of Staff, the Commander, U.S. Northern Command, and 
other relevant combatant commanders, coordinate with the 
Department of Homeland Security and Federal Emergency 
Management Agency to develop guidance and procedures for 
consistently involving state officials in planning, executing, and 
assessing exercises that incorporate relevant state-specific 
information, and 

• the Secretary of Defense direct NORTHCOM's Commander to develop 
a training plan for NORTHCOM headquarters staff on state emergency 
management structures and relevant issues related to working with 
civilian state and local emergency management officials. 

To improve NORTHCOM's involvement of interagency partners and states 
in its lessons learned and corrective action process and its management of 
corrective actions, we recommend that the Secretary of Defense direct: 



NORTHCOM's Commander to establish and publicize valid and easily 
accessible procedures for non-DOD exercise participants to submit 
observations relevant to NORTHCOM, such as placing a template on 
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NORTHCOM's publicly accessible Web site or DHS's Homeland 
Security Information Network, so that NORTHCOM officials have a 
clear, secure avenue to obtain observations and assess potential 
lessons that originate with its exercise partners; 

• the Chairman of the Joint Chiefs of Staff, in consultation and 
coordination with DHS, to either resolve information assurance issues 
so that the combatant commands, including NORTHCOM, can post 
Exercise Summary Reports with lessons learned and observations 
from NEP exercises on DHS's Lessons Learned Information Sharing 
system to make them easily accessible to interagency partners and 
states or establish an alternative method to systematically collect and 
share lessons learned; and 

• the Chairman of the Joint Chiefs of Staff to revise the joint lessons 
learned operating instruction to include procedures to ensure that 
appropriate corrective actions are implemented and verified in a 
subsequent exercise or operation before being closed and that the 
reasons for closure are documented. Possible procedures might be 
adding a verification checkbox on JLLIS's issue management page or 
requiring that the directorates and subordinate commands within the 
combatant commands provide a status report when a correction is 
implemented and reobserved or closed for reasons other than re- 
observation. 

To improve NORTHCOM's ability to work with interagency partners on 
major national exercises and further achieve the objectives of the NEP, we 
recommend that the Secretary of Defense revise the instruction on DOD 
participation in the NEP and/or direct the Chairman of the Joint Chiefs of 
Staff to revise the operating instruction regarding DOD participation in the 
NEP to provide the general conditions under which the combatant 
commands are expected to follow the Homeland Security Exercise and 
Evaluation Program planning and documentation requirements or the 
DOD's Joint Training System should be modified for those civil support 
exercises. 



Agency Comments 
and Our Evaluation 



In comments on a draft of this report, DOD generally agreed with the 
intent of our recommendations and discussed steps it is taking or plans to 
take to address these recommendations. DOD also provided technical 
comments, which we have incorporated into the report where appropriate. 
DHS also reviewed a draft of this report and provided technical comments, 
which we have incorporated where appropriate. 
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In response to our recommendation that NORTHCOM develop guidance 
with specific criteria for postexercise documentation to allow the results 
and lessons learned of exercises to be reviewed and compared, DOD 
agreed that such information should be provided in a standardized format 
that can be easily accessed and understood by authorized organizations 
which might benefit from such knowledge. DOD cautioned that any 
actions in response to this recommendation must accommodate 
constraints regarding classified information. We agree that properly 
securing classified information is a critical responsibility and believe this 
can easily be accomplished without undermining the intent of the 
recommendation, which is to improve the consistency and completeness 
of formal exercise documentation and thereby its overall value. 

In response to our recommendation that DOD coordinate with DHS and 
FEMA to develop guidance and procedures for consistently involving state 
officials in planning, executing, and assessing exercises that incorporate 
relevant state-specific information, DOD agreed that better coordination 
for interfacing with state officials can be achieved. DOD also pointed out 
that NORTHCOM continues to expand its efforts to work through defense 
coordinating officers, existing state National Guard relationships, and 
FEMA regional headquarters partners to ensure that states are able to 
benefit from participation in DOD-sponsored exercises. However, DOD 
also said that while NORTHCOM has continuously engaged and 
encouraged state participation in NORTHCOM-sponsored exercises, the 
primary audience for such training is and must remain NORTHCOM. DOD 
also suggested that our recommendation has applicability to other federal 
interagency partners and that the issue should be addressed to the 
Exercise and Evaluation Sub-Interagency Planning Committee as a 
revision to the National Exercise Program Implementation Plan. As our 
report indicates, we agree that NORTHCOM has sought to engage and 
involve the states in its comprehensive exercise program. NORTHCOM 
plans for and conducts major exercises both inside and outside the 
construct of the National Exercise Program. Particularly for NORTHCOM- 
sponsored exercises focused on the command's civil support mission, the 
effective involvement of and interaction with state and other federal 
partners is a critical component of improving and maintaining 
NORTHCOM's preparedness. For NORTHCOM's participation in national- 
level exercises, the preparedness goals and objectives of all participants 
are equally important. We believe that in developing procedures to 
improve coordination with the states, DOD can (1) avoid situations where 
exercises meant to improve preparedness are not fully coordinated with 
the necessary partners; (2) capitalize on the structures and organizations it 
already has in place, such as the defense coordinating officers and 
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relationships with state National Guard headquarters; and (3) coordinate 
with DHS and FEMA to improve the military-civilian interface. With regard 
to the latter, the Exercise and Evaluation Sub-Interagency Planning 
Committee may indeed be one of the venues at which DOD can effectively 
coordinate with its interagency partners. 

With respect to our recommendation that NORTHCOM develop a training 
plan for NORTHCOM headquarters staff on state emergency management 
structures and relevant issues related to working with civilian state and 
local emergency management officials, DOD agreed and noted that 
headquarters training is required for all newly assigned NORTHCOM staff. 
Further, DOD noted that NORTHCOM sponsors three versions of its 
defense support of civil authorities seminar that are targeted to staff at 
different seniority levels. We agree that NORTHCOM has continued to 
improve the level of awareness and training it provides staff on the 
complexities of providing defense support to civilian authorities in the 
United States. However, this does not fully address our recommendation. 
While training on the general procedures of the national response 
framework, the nature of state-federal government relations, and DOD's 
proper role in providing military support to civil authorities is invaluable 
for NORTHCOM staff, we continue to believe that this should be 
supplemented by the kinds of state-specific information that would 
provide both exercise officials and all other staff with an understanding of 
the key differences between states. These differences are possibly as 
numerous as the number of states and play a role in all routine 
interactions between the individual states and DOD officials as well as for 
effective coordination for exercise planning and coordination during a 
natural disaster or some other no-notice incident requiring defense 
support to civil authorities. 

DOD agreed with our recommendation that NORTHCOM establish and 
publicize valid and easily accessible procedures for non-DOD exercise 
participants to submit observations relevant to NORTHCOM, such as 
placing a template on NORTHCOM's Web site or DHS' Homeland Security 
Information Network, so that NORTHCOM officials have a clear, secure 
avenue to obtain observations and assess potential lessons that originate 
with its exercise partners. DOD indicated that collecting exercise 
information from all perspectives would provide additional opportunities 
to improve NORTHCOM's ability to accomplish its mission tasks. 

DOD also agreed with our recommendation that it work with DHS to 
either resolve information assurance issues so that NORTHCOM can post 
Exercise Summary Reports with lessons learned on DHS' Lessons Learned 
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Information Sharing system or establish an alternative method to 
systematically collect and share lessons learned. DOD cautioned that 
while wide dissemination of information approved for release would be of 
great benefit to homeland security entities it continues to adhere to the 
Joint Training System and cannot mandate that DHS alter its Lessons 
Learned Information System to make accommodations. DOD also noted 
that it has procedures in place to allow specifically cleared individuals 
from outside DOD access to information contained in Exercise Summary 
Reports. We agree that DOD cannot mandate alterations to the Lessons 
Learned Information System. We also agree that the Joint Training System 
should remain the chief guidance for the conduct of DOD exercises. 
However, we continue to believe that in working with DHS on the proper 
level and mode of information sharing, DOD may be able to improve the 
dissemination of relevant exercise-related information to all appropriate 
officials. 

DOD agreed with our recommendation that the Chairman of the Joint 
Chiefs of Staff revise the joint lessons learned operating instruction to 
include procedures to ensure that appropriate corrective actions are 
implemented and verified in a subsequent exercise or operation before 
being closed and that the reasons for closure are documented. DOD 
indicated that the Chairman of the Joint Chiefs of Staff Instruction 
3150. 25D could be expanded to provide more guidance and the Joint 
Lessons Learned Information System could be updated to provide a 
technological solution to address the issue once the process and 
procedures are in place. DOD also indicated that the process of verifying 
corrective action and closing issues will become more effective with the 
modifications it outlined in response to the recommendation. 

In response to our recommendation that DOD revise guidance on DOD 
participation in the National Exercise Program to provide the general 
conditions under which the combatant commands are expected to follow 
the Homeland Security Exercise and Evaluation Program planning and 
documentation requirements or the DOD's Joint Training System should 
be modified for those civil support exercises, DOD recognized the 
importance of ensuring effective interaction with interagency partners for 
homeland security-related exercises. However, DOD noted that the 
National Exercise Program Implementation Plan contains language placed 
there at DOD's insistence that establishes a process to resolve doctrinal 
differences during exercise planning. DOD indicated that together with 
provisions in the implementation plan establishing the administration, 
scope, and hierarchy of multiagency homeland security exercises and the 
5-year National Exercise Program schedule, this should address our 
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recommendation. DOD further noted that The Joint Training System 
remains the Secretary of Defense's guidance on DOD exercises and that 
the National Exercise Program Implementation Plan stipulates that 
individual department or agency exercise programs should not be 
replaced. We agree that the Joint Training System is and should be DOD's 
primary guidance for ensuring that DOD components train and exercise 
according to standards. However, because interagency exercises are 
becoming an ever larger part of the national preparedness effort, and to 
the extent that effective exercise planning is bolstered by common 
procedures, our recommendation is intended to help DOD clarify for its 
components the circumstances under which the specific planning and 
documentation requirements for the Homeland Security Exercise and 
Evaluation Program can be followed without detriment to DOD's high 
training and exercise standards or compromise of the Joint Training 
System. 

DOD's written comments are reprinted in appendix III. 



We are sending copies of this report to the Secretary of Defense, Secretary 
of Homeland Security, and other interested parties. In addition, the report 
will be available at no charge on GAO's Web site at http://www.gao.gov. 

If you or your staff have any questions about this report, please contact me 
at (202) 512-5431 or dagostinod@gao.gov. Contacts points for our Office of 
Congressional Relations and Public Affairs may be found on the last page 
of this report. Key contributors to this report are listed in appendix IV. 




Davi M. D'Agostino 
Director, Defense Capabilities 
and Management 
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Appendix I: Scope and Methodology 



In conducting this review, we generally focused our scope on U.S. 
Northern Command's (NORTHCOM) large-scale exercises 1 conducted 
since Hurricane Katrina made landfall in August 2005. To determine the 
extent to which NORTHCOM's exercise program is consistent with 
Department of Defense (DOD) training and exercise requirements and 
includes relevant exercise partners, we evaluated NORTHCOM's 
compliance with exercise reporting and documentation requirements 
established in DOD and command guidance. We reviewed available 
guidance to determine requirements for timing, format, and content. We 
also compared these requirements with guidance contained in the 
Department of Homeland Security's (DHS) Homeland Security Exercise 
and Evaluation Program documentation. We reviewed exercise 
documentation for all large-scale exercises the command performed since 
it was established in 2002 to determine the extent to which the command 
complied with the guidance. 

We also performed an assessment of the experiences and level of 
participation from some interagency organizations and states in 
NORTHCOM's large-scale exercises. We initially met with Nevada officials 
who participated in a NORTHCOM exercise prior to Hurricane Katrina — 
Determined Promise 03 — to provide context to the extent that changes 
may have been made to NORTHCOM's exercise program and help develop 
our state selection methodology. We selected a nongeneralizable sample of 
six states based on the extent to which they have participated in major 
NORTHCOM exercise since Hurricane Katrina and the varying scenarios 
of the exercises. The states we selected played a major role in 
NORTHCOM exercises by having a portion of the exercise conducted in 
their state and having various state agencies and officials participate. 
States we selected include Arizona, California, Michigan, Oregon, Rhode 
Island, and Washington. We met with representatives from each state's 
emergency management organization and state national guard. Because of 
the methodology selected, the resulting data and information from these 
state visits could not be projected to make assumptions about the rest of 
the states and what they may experience exercising with NORTHCOM. We 
also met with officials from three Federal Emergency Management Agency 
(FEMA) regional offices that had exercised with NORTHCOM in three of 
the last six large-scale exercises. We also interviewed officials from the 



^or purposes of this report, large-scale exercises are those involving multiple agencies and 
organizations, venues, and events, such as NORTHCOM's Vigilant Shield and Ardent Sentry 
exercises. NORTHCOM generally conducts two large-scale exercises annually. 
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Appendix I: Scope and Methodology 



Office of the Assistant Secretary of Defense for Homeland Security and 
America's Security Affairs, Joint Staff, and NORTHCOM with knowledge 
of and experience with NORTHCOM's training and exercise program. 

To determine the extent to which NORTHCOM is using lessons learned 
during exercises to improve mission preparedness, we reviewed DOD, 
NORTHCOM, and DHS National Exercise Program (NEP) guidance for 
recording, tracking, and managing lessons learned and assessed 
NORTHCOM's management of exercise observations and issues identified 
in several of NORTHCOM's large-scale exercises since Hurricane Katrina 
in 2005. We interviewed NORTHCOM, Joint Staff, and FEMA officials 
regarding the various lessons learned management systems, and how 
interagency and state access to these systems can be accomplished. We 
also spoke with an official in the General Services Administration 
regarding the types of federal personal identification verification cards 
used by DOD and other federal departments and agencies to access 
government computer systems. In reviewing the management of 
NORTHCOM's lessons learned program we identified and reviewed all 
unclassified exercise observations from its last six large-scale exercises 
that had been activated in NORTHCOM's area of the Joint Lessons 
Learned Information System (JLLIS). Our review of the records in JLLIS 
entailed determining each record's status (open or closed), its type (issue 
or lesson learned), and each record's disposition after NORTHCOM staff 
have acted on these records to respond to the issues or lessons learned 
documented. Based on our review, we generally placed these records into 
one of several categories: open; closed, nonconcur; issue closed with 
reobservation; issue closed with no reobservation; and lesson learned. In 
addition we reviewed several records that had been merged with other 
original records because each related to the same issue; however, the 
original record for that issue was not part of our universe. Therefore, 
without reviewing the lead record the merged records lacked sufficient 
information regarding their disposition and that condition became another 
category. 

Finally, to determine the extent to which NORTHCOM is integrating its 
training and exercises with the NEP we reviewed DOD, NORTHCOM, and 
Department of Homeland Security guidance to identify any differences in 
exercise planning and documentation between DOD's guidance and that 
for the NEP. We used that analysis to determine under what conditions 
NORTHCOM should apply standards related to the NEP, and how DOD 
and its subordinate commands should participate in the NEP tier I or II 
exercises. We reviewed NORTHCOM documentation from two major 
national exercises conducted during fiscal year 2008 to determine the 
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extent to which NORTHCOM employed the guidance from the Homeland 
Security Exercise Evaluation Program. We determined that national 
exercises that are operations-based exercises in that they involved the 
deployment of personnel would be the best candidates for evaluating 
NORTHCOM's participation in such exercises. We also interviewed state 
emergency management and National Guard officials from six states that 
have exercised with NORTHCOM since 2005, to understand the extent to 
which NORTHCOM is integrating its exercise planning and conduct with 
the interagency as well as various state governments. 

In addressing our objectives, we reviewed plans and related documents, 
obtained information, and interviewed officials at the following locations: 

• NORTHCOM Headquarters, Peterson Air Force Base, Colorado 
Springs, Colorado 

• Joint Forces Command, Joint Warfighting Center, Suffolk, Virginia 

• The Office of the Secretary of Defense, Washington, D.C. 

• The Joint Staff, Washington, D.C. 

• U.S. Army North, Fort Sam Houston, San Antonio, Texas 

• National Guard Bureau, Arlington, Virginia 

• Department of Homeland Security, Washington, D.C. 

• U.S. Coast Guard, Atlantic Area, Portsmouth, VA 

• FEMA's National Preparedness Directorate, Washington, D.C. 

• FEMA Region 1, Boston, Massachusetts 

• FEMA Region 9, Oakland, California 

• FEMA Region 10, Bothell, Washington 

• General Services Administration, Washington, D.C. 

• Arizona Division of Emergency Management, Phoenix, Arizona 

• Arizona National Guard, Joint Force Headquarters, Phoenix, Arizona 

• California Emergency Management Agency, Sacramento, California 

• California National Guard, Joint Force Headquarters, Sacramento, 
California 

• Michigan State Police, Emergency Management and Homeland 
Security Division, Lansing, Michigan 

• Michigan National Guard, Lansing, Michigan 

• Nevada State Division of Emergency Management, Carson City, 
Nevada 

• Nevada National Guard, Joint Force Headquarters, Carson City, 
Nevada 

• Rhode Island Emergency Management Agency, Cranston, Rhode Island 

• Rhode Island National Guard, Joint Force Headquarters, Cranston, 
Rhode Island 

• Oregon Military Department, Office of Emergency Management, 
Salem, Oregon 
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• Oregon Military Department, National Guard Joint Force 
Headquarters, Salem, Oregon 

• Washington Military Department, Emergency Management Division, 
Camp Murray, Washington 

• Washington National Guard, Joint Force Headquarters, Camp Murray, 
Washington 

We conducted our review from June 2008 to September 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 
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Appendix II: Comparison of DOD's Joint 
Training System with DHS's Homeland 
Security Exercise and Evaluation Program 



We reviewed the time lines and milestones for developing exercises found 
in the Department of Defense's Joint Training System and U.S. Northern 
Command's (NORTHCOM) implementing guidance and compared them 
with the Department of Homeland Security's Homeland Security Exercise 
and Evaluation Program guidance to determine the similarities and 
differences between them. We used the guidance associated with 
operation-based exercises rather than discussion-based exercises to 
present the full spectrum of Homeland Security Exercise and Evaluation 
Program processes and planning events. 



Table 8: Comparison of DOD Joint Training System and Homeland Security Exercise and Evaluation Program 



DOD Joint Training System and NORTHCOM Exercise 

Program Homeland Security Exercise and Evaluation Program Guidance 

Joint Event Life Cycle Homeland Security Exercise and Evaluation Program Exercise Cycle 



NORTHCOM uses the Joint Events Life Cycle to design, plan, 
and execute exercises. 

Consists of 5 stages: design, planning, preparation, 
execution, and evaluation. 

Is a flexible guide that can be modified to apply to various 
types of events 

Can span a period of 1 2 to 1 8 months for exercise 
development. 



Homeland Security Exercise and Evaluation Program policy and 
guidance is the doctrine for the development of National Exercise 
Program exercises. 

Employs 5 phases, which are collectively known as the exercise 
cycle: foundation, design and development, conduct, evaluation, 
and improvement planning. 

Planning time line for the national level exercise will begin 
approximately 15 months prior to exercise conduct. 



Concept Working Group 



Concept and Objective Meeting 



Time frame: 1 2 to 15 months prior to start of exercise. 

Purpose: 

Commences start of design stage. 

Conduct initial planning to develop the candidate 
scenario setting, determine funding constraints, and 
define training requirements. 

Result in the validation of the exercise purpose, exercise 
objectives, identification of the training audience, 
intelligence concepts, draft exercise event flow, planning 
milestone, and draft initial planning conference agenda. 



Time frame: Minimum 6 months prior to start of exercise, lasts 2 to 4 
hours. 

Purpose: 

Marks the start of the planning process and can be conducted in 
conjunction with the initial planning conference. 

Identify the type, scope, objectives, and purpose of the exercise, 
as well as help planners identify, among other things, exercise 
planning team members. 

Reach agreement regarding exercise type, scenario, capabilities, 
tasks, objectives, the target exercise time frame, and the date and 
time of the next planning conference. 
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Appendix II: Comparison of DOD's Joint 
Training System with DHS's Homeland 
Security Exercise and Evaluation Program 



DOD Joint Training System and NORTHCOM Exercise 
Program 



Homeland Security Exercise and Evaluation Program Guidance 



Initial Planning Conference 



Initial Planning Conference 



Time frame: 10 to12 months prior to start of exercise, lasts 3 
to 4 days. 

Purpose: 

First major planning conference in the Joint Events Life 
Cycle process. 

Identify levels of participation, develop deployment and 
logistic requirements, validate the exercise milestones, 
develop outline of the exercise plan and other planning 
documents, determine modeling and simulation 
requirements, and develop the mid-planning conference 
agenda. 



Time frame: Minimum 6 months prior to start of exercise, lasts 3 to 6 
hours. 

Purpose: 

Begins planning process. 

Determine exercise scope by gathering input from the exercise 
planning team, design requirements and conditions (such as 
assumptions and artificialities), objectives, extent of play, and 
scenario variables (such as time, location, hazard selection). 

Used to develop exercise documentation by obtaining the 
planning team's input on exercise location, schedule, duration, 
and other relevant details. 



Mid-Planning Conference 



Mid-Term Planning Conference 



Time frame: 7-8 months prior to start of exercise, lasts 
approximately 3 days. 

Purpose: 

Develop a draft exercise plan, scenario, modeling and 
simulation architecture, after-action review concept of 
operations, detailed augmentation requirements. 

Update milestones and identify funding issues. In 
addition, workgroups should have master scenario 
events lists identified with an understanding of who is 
responsible for completing one. 



Time frame: Minimum 3 months prior to start of exercise, lasts 
approximately 1 day. 

Purpose: 

Discuss and determine exercise organization and staffing 
concepts, scenario and time line development, scheduling, 
logistics, and administrative requirements. 

Review draft documentation (such as scenario, exercise plan, 
controller and evaluator handbook, master scenario events list). 

Conduct a walkthrough of the proposed exercise site at the end of 
the conference. 



The Master Scenario Events List Development Conference The Master Scenario Events List Conference 



Time frame: Conducted before, after, or in conjunction with 
the mid-planning conference. 

Purpose: 

Forum for the initial development of detailed scenario 
events and the master scenario events list — a detailed 
narrative description of the exercise scenario in 
chronological sequence. 



Time frame: Conducted separately or combined with mid-planning 
conference or final planning conference; length varies from 4 to 8 
hours. 

Purpose: 

Focuses on developing the master scenario events list — a 
chronological list that supplements the exercise scenario with 
event synopses; expected participant responses; capabilities, 
tasks, and objectives to be addressed; and responsible 
personnel. 
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DOD Joint Training System and NORTHCOM Exercise 

Program Homeland Security Exercise and Evaluation Program Guidance 

Final Planning Conference Final Planning Conference 



Time frame: 3 months prior to start of exercise. 
Purpose: 

Provide a last check of the status of the planning and 
preparation for exercise and to finalize, publish, and 
distribute key exercise plans and documents (such as 
exercise plan, supporting plans, scenario, after-action 
review collection management plan, exercise manning 
status, control plan, exercise battle rhythm, and modeling 
and simulation status). 

Finalize the construct of the exercise with conference 
representatives. 

Master Scenario Events List Synchronization Conference 



Time frame: Minimum 6 weeks prior to start of exercise, lasts 
approximately 1 day. 

Purpose: 

Final forum for reviewing exercise processes and procedures. 

No major changes to the design or scope of the exercise, or its 
supporting documentation, should take place. 

Ensures that all logistical requirements have been met, all 
outstanding issues have been identified and resolved, and all 
exercise products are ready for printing. 



See master scenario events list conference above. 



Time frame: Held shortly after the final planning conference 
and at least 3 weeks prior to start of exercise. 

Purpose: All events and implementers are reviewed for 
timing, realism, and completeness. 

Tabletop Exercises and Senior Leadership Seminars Tabletop Exercises and Seminars 



Time frame: Prior to start of exercise. 
Purpose: 

Tabletop exercises are discussion-based exercises involving 
key personnel discussing hypothetical scenarios. 

Facilitating understanding of concepts and processes, 
and identifying strengths and shortfalls. 

Provide an opportunity to address new or side issues, or 
work out processes prior to the start of the exercise. 

Senior leadership seminars provide senior leaders (flag 
officer, general officer, and senior executive service level) the 
opportunity to discuss lessons learned from previous 
exercises and the upcoming exercise. 



Time frame: Not specified. 
Purpose: 

Tabletop exercises involve key personnel discussing hypothetical 
scenarios in an informal setting. 

Used to assess plans, policies, and procedures or to assess the 
systems needed to guide the prevention of, response to, and 
recovery from a defined incident. 

Seminars are informal discussions, unconstrained by real-time 
portrayal of events and led by a presenter. 

Orient participants to, or provide an overview of, authorities, 
strategies, plans, policies, procedures, protocols, response 
resources, and/or concepts and ideas. 



Event Tests 



Exercise Setup 



Time frame: Prior to start of exercise. Timeframe: Prior to start of exercise. 

Purpose: Conducted to ensure the modeling and simulation . Exerc j se panning team assigned to setup should visit the 
architecture is sufficient to meet exercise requirements. exercise site at least 1 day prior to the event to arrange the room 

and test audio/video equipment. 

On the day of the exercise, planning team members should arrive 
several hours before start of exercise to handle any remaining 
logistical or administrative items pertaining to setup, and to 
arrange for registration. 



Page 73 



GAO-09-849 Homeland Defense 



Appendix II: Comparison of DOD's Joint 
Training System with DHS's Homeland 
Security Exercise and Evaluation Program 



DOD Joint Training System and NORTHCOM Exercise 
Program 

Communications Exercise 

Time frame: Conducted approximately 2 weeks prior to start 
of exercise. 

Purpose: Confirm all exercise NORAD and NORTHCOM staff 
and external battle staff members have access to and are 
proficient with all required communications systems and 
collaborative tools. 

Exercise Execution 



Homeland Security Exercise and Evaluation Program Guidance 



See Exercise Setup above 



Exercise Conduct 



Time frame: Start of exercise. Time frame: Start of exercise. 

Executed per the exercise plan. • Includes setup and wrap-up activities. 

Depending on the exercise, it may only involve the • For an operations-based exercise, conduct encompasses all 
execution of a command post exercise or be greater in operations occurring between the designated start of the exercise 

scope and involve the simultaneous and/or concurrent and the end of the exercise, 

execution of other training events. 

Facilitated After-Action Review Hot Wash 



Time frame: After the end of the exercise. 

Purpose: A facilitated discussion held at the end of the 
exercise convened for the benefit of the training audience to 
examine actions and results during a training event. 



Time frame: Maximum 2 hours after end of exercise. 
Purpose: 

Provides opportunity to ascertain the level of satisfaction with the 
exercise, identify issues or concerns, and propose areas for 
improvement. 

Players complete and submit their Participant Feedback Forms 
during the hot wash. 

All evaluators take notes during play and hot washes for later 
compilation with other observations from their functional areas. 



After-Action Review 



Controller and Evaluator Debrief 



Time frame: May occur during and after the exercise. 

Purpose: An informal process designed by a commander or 
director to provide commanders direct feedback on the 
accomplishment of selected joint mission-essential tasks, 
conditions, and standards stated in terms of training 
objectives for the commander to evaluate training proficiency. 



Time frame: Maximum 1 day after end of exercise. 

Purpose: Exercise planning team leader facilitates this debrief, which 
provides each controller and evaluator with an opportunity to provide 
an overview of the functional area he or she observed and to discuss 
both strengths and areas for improvement. 



See facilitated after-action review. 



After-Action Conference 



Time frame: 5 weeks after end of exercise, lasts approximately 1 day. 
Purpose: 

Present, discuss, and refine the draft after-action report, and to 
develop an improvement plan. 

Solicit feedback and make necessary changes to after-action 
report. 

Source: GAO analysis of DOD and DHS documentation. 
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ASSISTANT SECRETARY OF DEFENSE 
2600 DEFENSE PENTAGON 
WASHINGTON, D.C. 20301-2600 



AUG 1 i 2009 



Ms. Davi M D'Agostino 

Director, Defense Capabilities and Management 

U.S. Government Accountability Office 

441 G Street, N.W. 

Washington, DC 20548 

Dear Ms. D'Agostino: 

This is the Department of Defense (DoD) response to the GAO draft report, GAO- 
09-849, "HOMELAND DEFENSE: U.S. Northern Command Has a Strong Exercise 
Program, but Involvement of Interagency Partners and States Can Be Improved," dated 
July 1 7, 2009 (GAO Code 351178). DoD concurs with four recommendations, and 
partially concurs with three recommendations. Our response to the recommendations is 
enclosed. 

Our point of contact for this action is Mr. Johnnie Wauchop, Office of the 
Assistant Secretary of Defense for Homeland Defense and Americas' Security Affairs 
(OASD (HD&ASA)), (703) 693-1968, or Johiinie.Wauchopfglosd.mil . 




P^ul N. Stockton 



Enclosure: 
As stated 
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GAO DRAFT REPORT - DATED JULY 1 7, 2009 
GAO 09-849(351178) 

"HOMELAND DEFENSE: U.S. Northern Command Has a Strong Exercise Program, but 
Involvement of Interagency Partners and States Can Be Improved" 



DEPARTMENT OF DEFENSE COMMENTS 
TO THE RECOMMENDATIONS 

RECOMMENDATION 1: The GAO recommends that the Secretary of Defense direct the U.S. 
Northern Command's (USNORTIICOM) Commander to develop guidance with specific criteria 
for post-exercise documentation, particularly the Exercise Summary Report as the official 
exercise record, including the content and format to be included in such reports that will allow the 
results and lessons learned of exercises to be easily reviewed and compared. 

POD RESPONSE: Partially Concur. DoD agrees that such information, when appropriate and 
approved for release, should be provided in a standardized format that can be easily accessed and 
understood by authorized organizations which may benefit from the knowledge acquired through 
our exercise activities. Any solutions must be able to accommodate constraints regarding 
classified information. 



RECOMMENDATION 2 : The GAO recommends that the Secretary of Defense, in consultation 
with the Chairman of the Joint Chiefs of Staff, the Commander, USNORTI ICOM, and other 
relevant Combatant Commanders, coordinate with the Department of Homeland Security (DHS) 
and Federal Emergency Management Agency (FEMA) to develop guidance and procedures for 
consistently involving state officials in planning, executing, and assessing exercises that 
incorporate relevant state-specific information. 

DOD RESPONSE . Partially Concur. DoD concurs that better coordination for interfacing with 
State officials can be achieved. However, USNORTHCOM must rely on State and local exercise 
planning representatives to accurately and thoroughly represent their State Emergency 
Management structure and their training objectives during the 12-month planning process leading 
up to each exercise. To this end, USNORTHCOM continues to expand its efforts to work 
through its Defense Coordinating Officers/Elements (DCOs/DCEs), its existing relationships with 
Slate National Guard forces, and with FEMA regional headquarters partners to ensure that States 
are also able to benefit from participating in DoD-sponsored homeland security related exercises. 

USNORTHCOM, throughout the history of its exercise program, has continuously engaged and 
encouraged the participation of State emergency management officials/structures in order to 
maximize available training opportunities for all stakeholders. Additionally, as noted in the GAO 
report, there are ample references to the amount of State "play" and the benefits that States have 
derived from USNORTHCOM-sponsored exercises. Ultimately, however, it must be 
acknowledged that the primary training audience in any USNORTHCOM-sponsored exercise is 
USNORTHCOM. 

Finally, the GAOs proposed recommendation has applicability to other US Government 
interagency partners, as well as DoD. As such, the issue should be addressed by the Exercise and 

1 
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Evaluation Sub-Interagency Planning Committee as a revision to the National Exercise Program 
(NEP) Implementation Plan. 



RECOMMENDATION 3: The GAO recommends that the Secretary of Defense direct 
USNORTHCOM's Commander to develop a training plan for USNORTHCOM headquarters 
staff on State emergency management structures and relevant issues related to working with 
civilian State and local emergency management officials. 

POD RESPONSE: Concur. Defense Coordinating Officers, who report to USNORTHCOM vi 
U.S. Army North (USARNORTH), are well-versed in their respective States' procedures. 
Additionally, the headquarters staff training specified in the recommendation, which has been in 
existence since 2002, is required for all newly assigned USNORTHCOM staff members. The 
training includes a full discussion of how States request and receive Defense Support of Civil 
Authorities (DSCA). Further, USNORTHCOM sponsors three versions of its formal DSCA 
seminar which are described as follows: 



• A three phase DSCA course that includes eight hours of online instruction, a one-week 
classroom seminar, and bi-monthly post-training e-mail updates on DSCA news items. 

• A DSCA Executive Seminar offered four times per year and tailored to senior official 
training audiences in DoD (including the National Guard), interagency organizations, 
Stale emergency management agencies, local governments, and non-governmental 
organizations. 



A soon-to-be available DSCA Senior Executive Seminar, a focused half-day session 
intended for Governors, State Adjutants General, and State Directors of Emergency 
Management. 



RECOMME NDATION 4 : The GAO recommends that the Secretary of Defense direct 
USNORTHCOM's Commander to establish and publicize valid and easily accessible procedui 
for non-DOD exercise participants to submit observations relevant to USNORTHCOM, such £ 
placing a template on USNORTHCOM's publicly accessible website or the DHS's Homeland 
Security Information Network, so that USNORTHCOM officials have a clear, secure avenue t, 
obtain observations and assess potential lessons that originate with its exercise partners. 

DOD RESPONSE: Concur. Collecting relevant exercise and training information from all 
perspectives will undoubtedly provide additional opportunities to improve USNORTHCOM's 
ability to accomplish its mission essential tasks. 



RECOMMENDATION S : The GAO recommends that the Secretary of Defense direct the 
Chairman of the Joint Chiefs of Staff, in consultation and coordination with DHS, to either 
resolve information assurance issues so that the combatant commands, including 
USNORTHCOM, can post Exercise Summary Reports with lessons learned and observations 
from the National Exercise Program (NEP) exercises on DHS's Lessons Learned Information 
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Sharing system to make them easily accessible to interagency partners and states or establish an 
alternative method to systematically collect and share lessons learned. 

POD RESPONSE ; Concur. DoD agrees that such information, when appropriate and approved 
for release, should be disseminated as widely as possible to those homeland security related 
entities which may benefit from the knowledge acquired through our exercise activities. 
Concurrence, however, should not be misconstrued as support for altering or departing from the 
Joint Training System and its components, specifically the Joint Lessons Learned Information 
System. Additionally, DoD cannot mandate that DHS alter its Lessons Learned Information 
System nor require DHS to commit resources to do so. There are procedures in place that allow 
specifically cleared individuals from outside DoD access to information contained in the Exercise 
Summary Reports. These manual procedures ensure that only appropriate and authorized 
information is offered outside the Department. 



RECOMMENDATION 6: The GAO recommends that the Secretary of Defense direct the 
Chairman of the Joint Chiefs of Staff to revise the joint lessons learned operating instruction to 
include procedures to ensure that appropriate corrective actions are implemented and verified in 
subsequent exercise or operation before being closed and that the reasons for closure are 
documented. Possible procedures might be adding a verification checkbox on the Joint Lessons 
Learned Information System's (JLLIS) issue management page or requiring that the directorates 
and subordinate commands within the combatant commands provide a status report when a 
correction is implemented and re-observed or closed for reasons other than re-observaticn. 

DOD RESPONSE: Concur. Chairman of the Joint Chiefs of Staff Instruction (CJCSI) 
3 1 50.25D provides policy and guidance for the Joint Lessons Learned Program and depicts the 
"revalidation" of issues to ensure lessons have been learned (Figure A-2, Joint Lessons Learned 
Program Process Overview). This discussion section could be expanded to provide more 
guidance and the JLLIS could be updated to provide a technological solution to address rhe issu< 
once the process and procedures are in place. 

Regarding internal tracking, JLLIS provides sufficient capability for directors and subordinate 
commands to effectively manage internal issues. JLLIS Tier I administrators can make local 
modifications to facilitate this process. The process of verifying corrective action and closing 
issues will become more effective with the suggested modification, continued education, and 
further maturation of JLLIS and the Joint Lessons Learned Program. 



RECOMME NDATION 7 : The GAO recommends that the Secretary of Defense revise the 
instruction on DOD participation in the NEP and/or direct the Chairman of the Joint Chiefs of 
Staff to revise the operating instruction regarding DOD participation in the NEP to provide the 
general conditions under which the combatant commands are expected to follow the Homeland 
Security Exercise and Evaluation Program planning and documentation requirements or the 
DOD's Joint Training System should be modified for those civil support exercises. 

DOD RESPONSE: Partially Concur. DoD recognizes the importance of ensuring the 
Department is sufficiently able to interact with our interagency partners at every level regarding 
the planning and execution of Homeland Security-related exercises. To this end, DoD has fully 
supported and, when appropriate, has directly assisted in the development of the NEP and its 
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underlying doctrine, methodologies, and technologies. Evidence of these efforts are readily found 
in the GAO report where the similarities between DoD exercise methodology and National 
Exercise Program doctrine are obvious. 

The NEP Implementation Plan (I-Plan) clearly delineates how DoD participates in the NEP 
Additionally, largely due to DoD's insistence, the 1-PIan contains language in paragraph 6 2 
establishing processes to resolve doctrinal differences during the initial stages of NEP exercise 
planning. Further, the NEP I-Plan establishes the administration, scope, and hierarchy of 
homeland security-related exercises involving multiple interagency entities. Combined with its 
associated NEP five-year schedule which includes USNORTHCOM homeland security-related 
exercises, the 1-PIan and the five-year schedule appropriately address the GAO's 
recommendation. 

It should be emphasized, however, that the Secretary of Defense retains sole authority for training 
and equipping the Department of Defense. The Joint Training System, as it is currently 
administered, constitutes the Secretary's guidance regarding this matter. This position is further 
reinforced in paragraph 4 of the NEP Implementation plan, which states "The NEP is intended to 
provide a framework for prioritizing and focusing Federal exercise activities, without replacing 
any individual department or agency exercise program". 
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